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(57) /^STRACT 

Disclosed is an iUnstrator- or painter-operated, computer- 
mediated system for creating binocular, stereoscopic images 
from 2-D and 3-D image elements created by the operator or 
imported into the system. Depth effects are synthesized by 
creating parallax offsets which are applied to the display 
positions of the various elements in the view presented to 
each eye. The magnitude of the offeet for each object is 
related to the depth of that object, the direction of the offset 
is related to which eye the view is intended for. A 3-D input 
device is described as a system component which may be 
used to control such a system. Additional techniques are 
disclosed whereby digital information so created may be 
displayed, broadcast, or utilized in a virtual reality system. 

109 Claims, 6 Drawing Sheets 
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FIGURE 3A : Vertical 
(Y) Mode - both balls 
move in the same 
direction, opposite to 
mouse movement . 
Similar to standard 
mouse mode. 



FIGURE 3B : Horizontal 
(X) Mode - both balls 
move in the same 
direction, opposite to 
mouse movement . 
Similar to standard 
mouse mode. 



FIGURE 3C : Twist 
(Third Axis) Mode — 
the two balls move in 
oppos i t e ( roughly ) 
horizontal directions. 
Unlike any standard 
mouse mC'de. 



07/24/2003, 



EAST Version: 



1.04.0000 



U.S. Patent Jul. 8, 2003 sheet 6 of 6 us 6,590,573 Bl 



LEFT 

EYE 

IMAGE 



NEAR AT FAR 

SCREEN 



FIGURE 4A 



RIGHT 

EYE 

IMAGE 




FIGURE 4B 



07/24/2003, EAST Version: 1.04.0000 



us 6,5! 

1 

INTERACTIVE COMPUTER SYSTEM FOR 
CREATING THREE-DIMENSIONAL IMAGE 
INFORMATION AND FOR CONVERTING 
TWO-DIMENSIONAL IMAGE 
INFORMATION FOR THREE-DIMENSIONAL 
DISPLAY SYSTEMS 

The instant iavention is a continuation-in-part of the 
applicant's application Ser. No. 07/435,487. This applica- 
tion is a continuation-in-part of applicant's application Ser. 
No. 07/227,403 now issued as U.S. Pat. No. 4,925,294 and 
also of application Ser. No. 07/006,291 now issued as U.S. 
Pat. No. 5,050,984 which was a continuation of Ser. No. 
06/601,091 which was a continuation-in-part of appUcation 
Ser. No. 06/492,816 now issued as U.S. Pat. No. 4,606,625. 
The instant invention is also a continuation-in-part of appli- 
cant's appUcation Ser. No. 07/077,916. All of the above 
referenced applications and patents are hereby incorporated 
in their entirety by reference. 

TECHNICAL FIELD 

The instant invention relates to an operator's interactive 
computer workstation for the conversion of two- 
dimensional motion pictures, video or other image informa- 
tion for three-dimensional display or viewing systems, as 
well as for the creation and integration of three-dimensional 
image information. As,such, this application relates to spe- 
cific details of additional embodiments and improvements to 
the systems described in and co-invented by the instant 
applicant as, U.S. Pat. No. 4,925,294 and the systems 
described in applicant's U.S. Pat. Nos. 4,606,625 and 5,050, 
984. 

PRIOR ART 

There exist a number of true three-dimensional, 
stereographic, binocular display systems, including: sepa- 
rate CRT displays for each eye, often routed through prisms 
or by mirrors, or mounted in a helmet sometimes with 
orientation sensors; and glasses with red and green ^or 
mutually polarized filters over the left and right eyes, for use 
with film or video displays. 

Many video displays, including home televisions, work 
with an interlaced raster image where each image frame is 
divided into two fields, with the even hnes displayed first 
followed by the odd lines (or vise versa). For US and 
Japanese systems the frame rate is 30/sec and the field rate 
is 60/sec; in Europe the rates are 25 and 50 respectively; for 
many industrial applications the rate is doubled to 60 and 
120 respectively; and, many other rates are available or 
possible. 

It is thus possible to form a stereoscopic 3-D display by 
creating a left and right image pair interleaved into the even 
and odd fields (or visa versa) of a video display, and viewing 
that display through LCD shuttered glasses which are flick- 
ered (180 degrees out of phase for each eye) in synchrony to 
the field rate of an interlaced video display, separating two 
* field' images from a single video * frame' signal. 

U.S. Pat. No. 4,952,594, of which the applicant is a 
co-inventor, describes a method for creating and modifying 
3-D program material derived from already existing 2-D 
motion picture films and video, and other information; 
including a system to produce such material. 

'Paint' and similar interactive programs permit an artist or 
operator to create 2-D paintings, drawings, illustrations and 
business, technical and industrial graphics on computer 
systems. 
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SUMMARY OF INVENTION 

The instant invention relates to techniques and devices for 
creating and/or delivering 3-D (stereoscopic, binocular, 

^ holographic, serial or otherwise) image information which 
may be derived from 2-D images or image elements con- 
verted to 3-D by the system, 3-D image elements created by 
the system, or integrated with pre-existing 3D image infor- 
mation created independent of the system. 

10 The primary system component is an interactive 3-D 
computer paint system which shares many elements with 
2-D computer paint systems, but which extends those ele- 
ments suitably to deal with 3-D image information. User 
interface elements such as menus, buttons, palettes, cursors, 

15 and information windows may be placed in the Z or depth 
direction (perpendicular to the plane of the display screen) 
as well as in X (horizontal) and Y (vertical); these elements 
may also have size or 'thickness' in the Z direction. In 
addition to image information (R, G & B or indexed colors) 

20 depth information may be stored with pixels in order to 
implement 3-D painting and processing functions. In this 
way, when further painting is to be performed at a particular 
pixel, the depth to be painted and the depth of what is 
already stored there may be compared; and, painting wotdd 

25 only be accomplished if the new information was closer to 
the viewer than (and should thus obscure) what was already 
there. 

Further, when painting, or other image manipulation or 
processing, functions are to be performed, they are per- 

30 formed on two separate images; one for the left eye and one 
for the right. However, the application of the function in 
each picture is ofiiset in the horizontal direction by the 
appropriate parallax offset. The two images may then be 
interleaved for 3-D display. Thus, many standard 2-D paint- 

35 ing and image processing algorithms may be applied, sepa- 
rately to each picture, without having to re-design or 
re -program them for interleaved 3-D images. 

Techniques are disclosed for preparing such images for 
display with anaglyphic, LCD -shuttered, or new types of 

^ flickering glasses; or for broadcast by dual channel video 
systems. In addition, it is disclosed how such information 
may be adapted for use with virtual reaUty systems. 

Additionally, system components are disclosed which 
permit the input, by a human operator, of more positional 
information than simply X and Y position, yet which may be 
constmcted by combinations of simple X-Y transducers or 
other input elements used in combination. 

BRIEF DESCRIPTION OF DRAWINGS 

50 

FIG. 1 depicts the use of a standard paint system and a 
specialized 'disinterleaving' imaging function to create ste- 
reoscopic images. 

FIG. 2 depicts a 'disinterleaved' 4x4 pixel brush. 

FIG. 3 depicts two-ball (3-D) mouse operational modes. 

FIG. 4 shows the parallax offsets for near, far and screen 
depth information in the left and right images of a 3-D paint 
system. 

^° SCOPE OF INVENTION 

The instant invention relates to an operator's interactive 
computer workstation for the conversion of two- 
dimensional motion pictures, video or other image informa- 
65 tion for three-dimensional display or viewing systems. It 
may also be employed for the creation and processing of 
new 3-D image information, and for the integration of either 
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type of image ioformation with three-dimensional image appropriate software; or, any combination of the two. 

information imported into the system. Similarly, a nimiber of electronic functions may be embod- 

For example (as described in U.S. Pat. No. 4,925,294), ied as either digital or analog circuitry; and, various display 

when converting a particular stiU frame of a 2-D motion technologies— such as photographic or other fihn, holo- 

picture to 3-D, gaps between image parts may develop. In 5 g^phic fihn or display, lenticular screens, HDTV, plasma 

the patented system, functions are provided that permit the Panels, stationary or moving LEDs or LCDs, etc.— may be 

operator to fill gaps in the converted motion picture by substituted for video displays. It is mtcnded throughout that 

borrowing image information from other parts of the frame ^^se and similar substitutions, combmations or hybndiza- 

or from other frames in the sequence, or to create 'fill-in ^ions are within the scope of the instant inventioQ, even 

image information' by utilizing functions similar to known though not specificaUy stated below. 

2- D 'computer paint systeoas'. Id the absence of a 2-D The term '3-D Computer Graphics' is often used to 
motion picture frame to convert, such functions may be descn*be the rendering of a 2-D projection of a 3-D or *solid' 
utilized to create new 3-D images. As such, the system may mathematical model onto the plane of the computer display 
be used as a *3-D Paint System' for creating non-computed screen. Such images can be quite sophisticated, taking 
(i.e., operator or 'hand' generated) 3-D paintings, technical 15 complex Ughting, shading, shadows and reflections into 
illustrations, business graphics, or other operator specified account, and the computational techniques for such render- 
graphics. In addition, such a system could also be used for ing are highly developed. However, such '3-D Graphics* are 
adding image information to existing true 3-D images, 3-D only in the computational sense and result in a single 
including 3-D photographs and computationally generated 'monocular' 2-D image. In contrast, as used herein, the term 
binocular 3-D graphics. For example, circles, arrows, scales, 20 3_d will generally be used to mean binocular, stereoscopic 
axes and text notes may be drawn 'within the 3-D space' of or holographic images which exhibit depth characteristics 
a 3-D video display, in much the same way that a marking when viewed. 

pen can be used to annotate on the surface of a standard 2-D jt is also possible to create two 2-D computer rendered 

photograph. images from 3-D models, such that the perspective view of 

The technologies described herein may also be used, ^ each corresponds to the view as seen by the left and right eye 

either in an illustrator operated system or more automatic of the viewer. Such image pairs may be displayed by true 

system, for purposes of converting 2-D medical images to 3-D display systems to achieve synthetic stereoscopic imag- 

3- D. In this way, traditional diagnostic and other medical ery. Such displays are sometimes used for entertainment 
images containing complex data may use the added depth purposes, but more often for industrial applications such as 
dimension to enhance visualization and increase compre- CAD or flight simulators. Such straight forward 'solid 
hensioD. model' rendering — even if made in stereoscopic pairs — is, 

The embodiments described herein employ LCD shut- gcneraUy, not being claimed as part of the instant invention, 
tered glasses, synchronized to the field rate of an interiaced Rather, the instant invention relates to an interactive system 
video display. Such a system may be used for both design which permits the operator to use 'pamterly' fimctions to 
and production of displayable and recordable end product on create 'true 3-D' images, exhibitmg a sense of depth, 
video. Similar systems may be designed around other ste- LasUy, discussion of vertical parallax wiU be omitted 
reoscopic or holographic display technologies; or, the sys- here, for the sake of simplicity and clarity. However — 
tem as described herein may be used by the operator to especially for objects close to the viewer, on the order of the 
design images that will later be produced for display, view- ^ interocular distance — noticeable vertical parallax or differ- 
ing or recording by any 3-D means cunently in use or later ential obscuring may come into effect. The same principles 
developed. Such variations are considered to be within the and techniques described herein for the horizontal case may 
scope of the instant invention. be applied in the perpendicular direction, as and if desired, 

Anumberofsystem components or techniques mentioned and are considered to be within the scope of the instant 

herein— ^uch as: stereoscopic systems of various kinds; 45 mvenuon. 

computer graphic devices, including frame buffers; interac- DESCRIPTION OF INVENTION 
tive computer graphic software systems such as pamt 

systems'; or, techniques, such as 'Z-buffering' — akeady As described in applicant's U.S. Pat. No. 4,925,294, 2-D 

exist and may be the subject of patents or other pubhcations. image elements may be relatively offset and combined with 

The literature on such elements is weU developed and the 50 other image information elements to create 3-D images. A 

reader is directed to that literature for additional details of computerized system for creating such images is described 

implementation. Although such components play a signifi- herein, 

cant role in the instant invention, it is not the purpose of this Three Display Schemes 

application to claim the invention of such basic components in one basic embodiment, the instant invention may be 
or techniques. Rather, the subject of the instant appUcation 55 implemented using standard, off-the-shelf hardware and 
are composite designs and processes, which employ and software components, with the addition of specialized soft- 
combine such existing components into novel and useful ware functions and the careful application of production 
systems or for new purposes, as well as numerous additional techniques. A standard 'paint' program, operating on a 
technical details which, in and of themselves, are novel. standard interlaced video computer display (frame buffer), 
A number of techniques will be described and/or claimed 60 may be used to convert separate 2-D image elements into a 
as being embodied in either digital hardware or software. It 3-D composite as follows. 

is often possible to simulate or mimic the functionality of In FIG. lA, the rearmost image element (10, a hollow 

special-puipose hardware, within an environment of general square in this example), called plane zero or PO, is brought 

purpose hardware, by the coding of control software. It is into the frame buffer by mnning a standard 'image load' 

intended that all of the techniques and functional structures 65 function of the paint software, or may be created using 

described herein may be embodied as either: specially computer paint ftmctions. In general, this PO element should 

constructed hardware; general purpose hardware with the contain image ioformation (even if it is only a 'blank' 
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background) to fill the entire area of interest; but, this area 
may be somewhat smaller than the entire display, as will be 
made clear below, to take into account left/right image shift. 

When PO is viewed through LCD shuttered glasses, the 
even/left image field and odd/right image field will register 
perfectly. There will be no differential shift or parallax and, 
thus, no depth effect. This 'neutraF depth plane will appear 
to be at the plane of the front of the CRT which will also be 
perfectly registered between the left- and right-eye views. 

At this point, a special software function is invoked. If the 
paint system software that is used does not permit calling a 
user specified outside function, it may be necessary to exit 
the paint software, run this special fimction, and then 
re-enter the paint program. This function will shift the 
even/left-eye field lines of the image in one direction by an 
amount SO and the odd/right-eye field lines in the other 
direction, generally by the same amount. See FIG. IB. 
(Note: the even/left — odd/right relationship may be as stated 
or reversed to odd/left even/right; only consistency is 
important.) This will cause the image plane to appear to 
move closer to (if SO is 'positive') or further from (if SO is 
* negative") the viewer, by an amount DO, due to the parallax 
between the two interleaved images. The larger SO is made, 
the more the image plane will appear to be moved in the 
direction perpendicular to the CRT plane, the *Z' direction. 
For the purpose of this discussion, the rearmost planes will 
be moved progressively further away from the viewer at 
each stage. 

In FIG. IC, the standard paint software is then used to 
load or paint a next-closer 2-D image plane, PI, (11, a 
vertical bar in this example) which may also have been 
derived by separating it from a 2-D motion picture frame, as 
in applicant's patent '425. This and later, more forward 
elements will, in general, be loaded with part of the image 
area * transparent' (or painted/created with * holes*) allowing 
PO or other relatively rearward planes to be seen through 
them. When images are stored as files, transparency is often 
denoted by storing a particular nimierical 'color' at those 
portions, often (0) for indexed or * color-mapped* images, or 
(0,0,0) for full-color, R-G-B images. 

In FIG. ID, the composite image, consisting of PO offset 
by SO and appearing to be DO behind PI, is now ofifeet in a 
similar fashion by the special software function, by an 
amount SI. Now PO is ofiket a total of SO-i-Sl, appearing DO 
behind PI and DO+Dl behind the face of the CRT; PI is 
offset by SI appearing Dl behind the face of the CRT. Note 
that, for certain circumstances, the apparent distances (Ds) 
may not, in fact, add linearly; but, the distances do 
accumulate, and the relative ordering of the planes is as 
described. 

ITie process is repeated and many 2-D image planes are, 
thus, stacked-up, back-to-front, to form a multi-planed, 3-D 
image when viewed through the flickering glasses. The 
back-to-front order guarantees that obscuring of rearward 
layers, which will be different in the left and right images 
due to parallax offsets, is done correctly. 

By progressively moving the composite image away from 
the viewer, or into the CRT display, the last, forward-most 
image plane will be at the front of the CRT and the full 
composite image will appear to be 'inside' the video display, 
similar to looking into a fish tank. By miming the odd/even 
differential shifting function one last time, by an amount 
equal to S0+S1+S2+ , . . -t-Sn-1, but in the opposite 
direction, the rearmost image plane may be restored to the 
plane of the CRT face. See FIG. IE. Thus, the composite 
image will then appear to come out of the CRT into 'room 
space'. This restoration process may be ninpart way and 
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arbitrarily set any position in the composite image (even 
between planes containing images) as the 'neutral' plane, 
without relative shift between the odd and even image fields. 

In shifting image elements left and right, some image part 
may 'fall off the edge' of the fi"ame buffer and be lost. As the 
image (which is spread, in opposite directions horizontally 
on alternate lines to create parallax, for alternating field 
stereo) is brought dose to the edge of the screen, the field for 
one eye or the other will 'fall off the edge' first. This will 
leave part of the image visible on only alternate lines, for a 
single eye, creating a ragged edge. Alternately, depending 
upon the details of the hardware and software used, the 
image information may 'wrap-around' and incorrectly 
re -appear at the opposite edge of the frame buffer. 

For this reason, the width of the area of the display used 
for imaging in this embodiment may best be made less, by 
the amount of the maximum shifts involved, than the maxi- 
TQum width of the display. To do this, the Z-Buffer may be 
initialized to a horizontally diminishing wedge to avoid 
images with ragged edges. 

If the minimum increment is 2 pixels (a minimum of one 
offeet in each direction) then to avoid ragged edges while 
painting, the last two usable pixel columns on each side must 
be avoided by the next farthest plane. This restriction 
continues, for further planes, loosing two additional on each 
side. This is true, starting fi:om the 'Oth' or neutral plane, for 
both into and out of the screen, although parallax is reversed 
between the two cases. 

Alternately, an extra wide frame buffer, providing non- 
visible pixels beyond the edges of the display area, may be 
employed and would permit the storage and restoration of 
those parts of image planes temporarily pushed off the edge 
of the display screen by the software shift function. Such a 
frame buffer would also avoid errors which could result from 
temporary wrap-around caused by shifting or unshifting the 
composite image. The AT- VISTA is an off-the-shelf display 
buffer for the IBM-PC that may be software configured to 
provide such capabilities. 

If the functionality of a paint system may be modified or 
if a new '3-D Paint System' is programmed, the "layer-shift- 
layer-shfft-layer ..." process may be incorporated into the 
program. 

Alternately, this process may be replaced by a more 
convenient functionality. This cumbersome process is 
required because a standard paint system 'paints' or per- 
forms its other imaging functions at the 'neutral' plane, 
where the odd and even image fields are not relatively 
shifted. In a preferred embodiment of the instant invention, 
the paint brush (and all other painting, graphic and imaging 
function such as: draw a line, circle, square, polygon, text or 
other graphic element; load, move, copy, or modify a portion 
of image; as well as all other traditional 2-D paint and 
graphic fanctions) would, itself, be programmed to have its 
odd and even image fields offset by a variable amount. 

FIG. 2 shows a small square 4x4 pixel brush. In 2A the 
brush is set for the zero or neutral plane with no relative 
offset of the odd and even lines of the brush image. In 2B the 
odd and even lines are offset for plane 1; in 2C for plane -1; 
in 2D for plane 2; in 2E for plane 20. 

Although much more convenient to use, this technique 
does have a drawback to implementation. The separated 
brush and interleaved left and right images require that any 
painting function be applied separately to the odd lines and 
even lines of the fi^ame buffer contents. Programming algo- 
rithms that are applied to non-continuous areas of frame 
buffer memory is both a conceptual and a computational 
comphcation. Additionally, the contents of many frame 
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buffers are manipulated by transferring rectangular areas of 7. A graphic of a knob, slider or other control icon may be 

the image to the computer for modification and then return- controlled by the artist; and additionally, 

ing the data to display memory. The overhead of initiating a 8. The computer may be programmed to compute a depth 

transfer is often very high compared to the transfer of each by: a depth value, depth tag, or other depth 

addilionalpixel. A 4x4 brush would require a single transfer 5 specification, stored with an image element; 

of 16 pixels back and forth. A separated brush as shown in interpolation, cither spatially over a firame, or tempo- 

FIG. 2E would require 2 transfers of 12 pixels (21 and 22) j-aUy between frames; by image analysis of any type; or, 

or a single rectangle containing 176 pixels (23). ^ described below, by consulting a depth or Z-buffer. 

A third embodiment that eUminates or rwiuces the above p^^^^ p^^^ ^^^^ q^^^ Movement 

problems can be miplemented in some con^^ ^-^^^ 1^^^ ^ -^^^^^ j^f^ right 

boards such as Truevision s AT-VISTA, which includes a interleaved for display, into the odd and even fields 

row table feature. These devices permit the programmed r c • J i_ - ij-iv 

specification of which memory line^^ll be displayed on the of a video frame, moving up or down by a smgle display hne 

screen for each display line. In that case the left and right the relaUonship bdtween which brush field is dis- 

images can each be placed in continuous rectangular blocks V^^y^^ ^^^^^ P^^^^^- ^hus, if, on even vertical cursor 
of the display *s memory. The display processor is 15 posiUons, the left brush is pamted mto the left image and the 

programmed, however, to display these two images in right brush is pamted mto the right image, on odd vertical 

interleaved fashion. The advantage is that all functions and cursor positions, the right brush is painted into the left image 

algorithms can then be programmed to operate on (two) and the left brush is painted into the right image. However, 

continuous rectangles of memory, rather than having to be the parallax shift also must be alternated or the brush will 
made more complex in order to work with two interleaved 20 flip in and out of the screen on alternate lines, 

views. Many standard, well-known and well-understood, If the above is termed Field Mode; a Frame Mode is also 

and *built in' functions can then be used as in standard possible. In Frame Mode, the cursor position may only move 

computer paint systems, including: drawing of lines, in multiples of two lines vertically and, thus, the relationship 

polygons, etc.; digital filters, anti-aliasing and other image between brush halves and picture halves is always pre- 
processing functions; color and pattern filling; etc. However 25 served. 

each function will have to be applied twice, once to each Three-Dimensional Image Processing Functions 
image, and all X-coordinates must be offset m each image by particular, it should be noted that image processing 
a parallax value appropriate to the depth plane at which the ^^^^^^^ ^-D painting functions, such as filtering, averaging, 
funcUon is to be apphed. . smearing, blurring or sharpening functions, usually are 
Another advantage of this method IS that different types of j ^ u i- j * *u • i • ™ * ™i 
J J**-.. jj'i r * u30 programmed to be apphed to the pixels in a rectangular 
mterleaved and non-mterleaved display formats can be • . -i 1. j * *u • 1 j 
accommodated for different display systems from the same 1'^^^' °' f f '5^^ shaped matte, or to those pixels iiiider 
frame buffer memory content, just by changing the table that ^^^P^ ^"f"' P™"ng' these 2-0 algo- 
, J. , 1 % u- u * nthms can be apphed to a smgle plane, or may be general- 
maps memory to display. For example, switching between , , , - , - ^ / t 1J 

1 p* / ' Uf/ e. *• « „ ^^ftuAA Z:^utu..^^ ized to be apphed to all the pixels m a 3-D rectangular solid, 

left/even right/odd configuration and a lert/odd nght/even , , . ^ ^ i • i -^x.- 

c 4- , - *u * *u * ui u u f* J 35 other arbitrarily shaped 3-D volume or, in particular, Within 

configuration only requires that the table entries be shifted , , / ^ J( , ^ t 

J u 11 ^*™u™j**u the volume of a 3-D brush shape. 

down or up by one Ime; also, a pad entry may be made at the ^ • • i • <_ . i * 

top or bottom, and one display line lost at the bottom or top. , J*,^ processing algorithms may be generalized to 

Similarly other display formats, such as those required for ^-D form as well; for example, blurrmg or softening 

the StereoGraphics brand of high frame rate alternating Actions are often implemented as algorithms that average 

, 1 , , 40 the pixel values (often with weightmg factors) wilhm a 

stereo displays can also be easily programmed. , i-j u, u 

This third scheme is the preferred embodiment of the given area which, when generalized would become aver- 

- . * • age all pixel values withm a given volume . Alteraalely, one 

instant invention. , , J^ - ^ ^ ^ • ■ 

Thr^e-Dimensional Positioning and 'Thickness' "J^^ ."PP^y f ditw"^ 7.V image Processing 

With the later embodiments described above, the artist algorithms to mdividual 2-D planes that are stacked m 3-D 

would be able to position the brush (or other imaging ^P="*' 7°""°g °° '"^ P«=l^^^' ^ ^^""^^"^ "fP'^ °\ 

function) along the Z (depth) axis as well as the X (left to range of depths. In tha case, when the computaUonal keme 

right) and Y (top to bottom) axes. In addition, the brush (or (neighborhood of pixels used to compute a parttcular pixel 

other imaging function) itself may have a depth 'thickness' ^^^'^) ^ ^ °''J<^'=' » P^°^»' f^'Tl 

or size, as weU as a position. non-standard options anse. For example, those pixels that 

The brush position, or size, along the Z-axis would be are m the kernel but not on a specified plane m question may 

displayed as a cursor, positionable in aU three dimensions, "^"""y ""^ computaUon; and may or may not be 

f , * 11 J u * ™„ * t 1 effected by the computation. Du-ectional algonthms, for 

and may be controlled by any appropriate computer control . , ^ , , - , 

. . 1 * J 1 J • 1 example bas rehef, may or may not be reversed in the two 

mechanism now m use or later developed, including: ^A^yi^^ uoo lk^u^l, iiAay vjl Li±aj nui ls. ^. , 

, , 1, eye views. Generally, based upon the depth at which a digital 

LA depth palette may be provided, much Iflce a standarf ^^^^^ ^ [ the center of the kernel will be 

color palette, that allows the artist to pick off a depth; horizontally displaced by the parallax offset differenUy for 

2. The above tnay be m the form of a depth ruler , as ^^^^ ^ -^^ ^^^^^^^ differential layering of image 
described below; elements in the two eye views will yield different results 

3. The artist may pick off a depth by pointmg at a ^^^^ digital filters are applied. Alternately, if the 'hidden* 
particular item or spot in the image; ^^^^^ ^ ^^^^ element on a particular plane are still 

4. A numerical value may be typed in at a keyboard; available to the computer, the digital filter kernel can be 

5. A pen with a variable pressure tip (rather than the applied to both the visible and obscured pixels of a plane, 
standard click-on tip) may be used to specify depth; Tlie Depth Buffer 

6. Other physical input transducers may be used, such as The above describes a convenient usable system for 
a knob, foot pedal, joystick, *data glove', 3-D input 65 creating and modifying stereoscopic images. However, the 
device as described below, or other control or position 3-D effect depends not only on horizontal parallax, but on 
sensor; or, the proper obscuring (which is subtly different for each eye) 
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of rearward image elecnents by forward image elements. If obscuring, only an order is required. Thus, the much cruder 

the visual cues of parallax and obscuring are inconsistent, resolution afforded by 8 bits can be used for the instant 

confusion results and the perception of 3-D is destroyed. By invention. For the purposes of this application, the use of a 

painting from back to front, one may guarantee that this depth channel will be referred to as a D -buffer 

aspect is properly handled; but, this is inconvenient and 5 In the simplest embodiment, when loading or creating a 

would interfere with the creative process by unduly restrict- layer, in addition to the differential offset function, for each 

ing the freedom to paint and modify an image in arbitrary pixel, the pixel value will only be used to replace the R-G-B 

order. An alternate embodiment, wherein depth information, values already stored for that pixel in the frame buffer if the 

in addition to display color information, is stored for image new pixel's depth is closer to the viewer and, therefore, 

pixels, takes care of this problem and is described below. lO should obscure that which is behind it. 

For many standard computer graphic display devices — For more flexibility, the depth range behind, equal to, and 

full color displays — memory planes are provided for storing in front of the selected depth for painting may be treated 

Red, Green and Blue values at each pixel; each pixel may separately and pixel-by-pixel replacement for each range 

have an independently specified color. Alternately, for color- may be individually permitted or not. Not all combinations 

mapped devices, a single number is stored at each pixel and 15 will make sense, but may still be used for effect, 

is translated into Red, Green and Blue values by using it as Additionally, individual depth planes, or ranges of depths 

an index into tables of Red, Green and Blue values— called may be protected from replacement or not. (See section on 

color look-up tables or color maps — ^with all pixels sharing "Protection",) 

a common index displayed as the same color. Interpretation of Depth Values 

In addition to the Red Green and Blue (or Index) infor- 20 For each possible D-buffer value, the value may represent 
mation for each pixel, it is common to store additional the actual depth. Alternately, a number may be associated 
information in other memory planes. One, three, eight, with each which indicates the actual depth which will be 
sixteen or more memory planes (corresponding to the num- used to offset the pixels of that plane; in effect a depth 
ber of bits per pixel) of additional information may be look-uptableor depth map, analogous to a color map. (Note 
provided, with eight being very common for moderate cost 25 that non-standard hardware or special software may be 
computer displays; often with 32 bits per pixel being required to implement some of the following techniques.) 
assigned as eight bits each for Red, Green, Blue and Alpha Thus, the 256 planes specifiable with an 8-bit D-buffer do 
(a non-displayed channel). The Alpha channel is tradition- not need to be evenly spaced, and may be positioned with 
ally xised for a number of purposes which can include: arbitrary resolution in the depth direction. When D-buffering 
overlay information (e.g., text or menus); video keying or 30 is to be done on a point-by-point basis for generally shaped 
(soft) matting of the computer display signal with another objects, rather than for just parallel planar elements, non- 
signal; masking off parts of the display from further pro- linear spacing may be particularly useful, 
cessing; temporary storage of items such as menus, icons or In general, this approach is useful and ehminates much of 
palettes — or, of the parts of images obscured by those items; the reason why 16 bits are often considered as required for 
and, pixel transparency. 35 Z-buffering. Sixteen bits are usually required because the 

For computer rendering of 2-D images of '3-D solid' (but space between 'nose close' and (for all intents and purposes) 

not stereoscopic) mathematical models, depth information is infinity needs to be divided into many segments if they are 

sometimes stored for each pixel to facilitate the proper to be of imifonn size. If too few segments are used, an 

display of many intersecting objects. Rather than trying to aliased or chunky effect will result when trying to effect 

compute the correct intersection of diverse objects, the 40 intersections between objects. 

pixels of each object calculated are only put into the display For example, if working from the screen plane into room 
if they are closer to the viewer than the equivalent pixels space, the relative left and right shift required for two objects 
already contained in the frame buffer, thus automatically separated by a few inches will be very great if the two 
creating the proper intersection between, and obscuring of, objects are close to the viewer; but, the left and right shift 
objects without their direct computation. Such a depth 45 required for two objects separated by a few inches will be 
memory plane is often called a Z-buffer. It is most often 16 virtually identical if the two objects are far from the viewer, 
or more bits deep, in order to provide enough resolution (2 Therefore, to correctly correspond to roughly imiform par- 
raised to the 16th power, or roughly 64 thousand divisions) allax ofkets (the S shifts) between consecutive D-values: 
for the proper mathematical placement of objects in the close (to the viewer) consecutive D-values may be separated 
computer's simulated 3-D (but still displayed as 2-D) envi- 50 by inches or even fractions of inches; consecutive mid-range 
ronment. values may be separated by several inches to feet; and, 

While a 16-bit buffer may be used in the embodiment of consecutive far-range values may be separated by many feet 

the instant invention under discussion, an eight-bit Alpha in 'scene space'. Uniformly spacing by equal 1/D values in 

channel may also be used to store depth information to great the D direction is one way that may be used to advantage for 

advantage, even though eight bits provide a resolution of 55 simulating true 3-D perspective. In this way, a relatively 

only 256 values. Eight bits of non-display memory is small number of possible D values may be used effectively, 

routinely available for computer paint systems while 16 So far, only planes parallel to the screen have been 

extra bits is not. A detailed discussion follows. discussed. In addition, as described in U.S. Pat. No. 4,925, 

While 256 divisions may not be sufficient to specify the 294,certain stereographic elements will not be in planes 

depth of a particular object in a simulated mathematical 60 parallel to the CRT faceplate. They may be tilted planes or 

space, in the instant invention 2-D image layers are often 3-D * sculpted' elements. In that case there may be several 

stacked up and differentially shifted, to create a stereo- planar elements with most of the D-values reserved for 

graphic display. These 2-D image layers may be ordered and non-parallel or non-planar elements. As an example, if three 

each assigned a depth or D value. This D value does not need depth values were reserved for three background planar 

to represent an exact depth value, just an ordering. The 65 elements, and three more for three for foreground planar 

differential odd/even field left/right offiscts will have to be element, that would still leave about 250 values for tiltcd- 

calculaled correcdy for an exact depth value; but for planar or sculpted elements. 
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Another embodiment provides additioQal flexibility. further from the viewer. That, in turn, may depend on the 

Assume that a D-buffcr containing 8 bits per pixel is focal length of a virtual camera or an actuaJ camera used to 

available, although the same basic idea could be used for photograph a flat image to which depth is to be added, 

other numbers of bits per pixel. The 8 bits will be split into Image elements that are planar, but tilted with respect to 

two segments: the upper or most significant 3 bits (providing 5 the face of the display, may be handled as thick or sculpted 

8 possible numbers 0 through 7) will be reserved as a 'tag'; elements as described above. However, particularly for tilted 

the lower or least significant 5 bits (providing 32 possible planes that extend over a wide range of depth values, the 

numbers 0 through 31) will.be used for a 'value'. limited number of possible values may produce a series of 

In this example, tag 0 wifl be reserved for planar objects strips parallel to the face plate, rather than a smooth plane, 

parallel to the screen. For planar elements the value wiU Such planes may be marked as such with their tag and 

represent the depth of that plane and can be used, as is, for handled as a special case. By storing positions for the edges 

obscuration purposes. However, for flexibility and added or corners of a rectangle, and applying the parallax rules to 

precision and, in particular, for interaction with other non- those positions, a pair of continuous smooth image elements 

parallel-planar elements, the actual depth value used to may be created for the left- and right-eye views by interpo- 

create parallax may be stored in a look-up table. Each of the ^5 lating between those edges or comers. Because of the simple 

32 entries in that table would be indexed to one of the 32 geometry associated with tilted planes, a precise depth for 

possible values for planar entries as described above. The each pixel may be easily calculated for the purposes of 

depth assigned to a particular pixel would thus be derived by proper obscuring with other image elements stored with 

using the value stored at that pixel as an index into the depth D-values at each pixel. By using a matte (perhaps stored in 

table. Thus, all the pixels associated with a planar element unused value portion, or stored as a separate raster 

conld be moved by changing a table entry rather than by image element), or by designating one 'transparent color', 

changing the value at each pixel. More than one planar the image element associated with the tilted plane need not 

element could thus be assigned the same depth. If overlap be rectangular, but may be given an arbitrary outline. Those 

occurs, then an ambiguous situation results. The tag num- pixels on the plane, but outside the matte, would thus be 

bers could be used to art)itrate the conflict. Alternately a ^5 ignored for both display and obscuring, 

second table (an 0-look-up table) could be constructed to Horizontal Resolution 

store ordering or obscuring information to be used all the Video displays in most computers, and the data files 

time, or only in conflict situations. In general two separate meant to be displayed thereupon, are often in the range of 

'depth' designations can be maintained: one, a 'distance' 512 pixels horizontally by 486 Hnes vertically. Other often 

value would be used to determine parallax offsets; the other, 33 used horizontal resolutions are 640, 768 (or 756, with six 

an 'ordering' or 'priority' value wotild be used to determine null, black or transparent pixels on each side) or even 1024 

which objects obscure other objects. or 1280. Straightforward embodiments use offsets on pixel 

Tags 1 through 7 would each be reserved for a tilted or boundaries and, therefore, the minimum horizontal paraUax 

'thick' image element. In these cases, the 32 possible values shift between planes adjacent in the depth direction is one 

could be interpreted differently than as described above, 35 pixel (usually two pixels, one each in opposite directions for 

with several alternatives: the left and right images) and, particularly at the lower end 

1. The 32 values associated with a particular tag could of t^iese horizontal resolution figures, this does not permit 
themselves be interpreted as actual depth values. separate planes or parts of objects to be closely spaced. 

2. The 32 values associated with a particular tag could The current embodiment of the system utilizes the 
themselves index a table of actual depth values (as 40 AT- VISTA display board which can be configured at many 
desCTibed above), providing an image element whose ^^^'^^^ resolutions, including a relatively high 1008 pixels 
individual pixels were distributed at up to 32 depths. A horizontally (which may be increased to a nominal 1536). 
second order table could also be used. ^hen importing images painted, rendered or scanned at 

3. Tlie 32 values could index a table of relative depth ^^^^ jisu^l resolutions, data pixels are repeatedly loaded 
values, providing (probably fairly small) depth ofikets 45 higher resolution display s memory Io.cations.,For 
from a positioning depth value also associated with the example, with 5 12.pixel resolution nnages into a 1536 pixel 
tag. Hie depth value for each pixel of the image >?^de screen, each pixel would be 

element is calculated by adding the offset table entry ^^tF'^^LlT^''T T^^^ '^""^ ^ .'^ \ 

associated with the value for that pixel, from the table ^^^^ 1024.pixel resoluUoD images each data pixel is 

associated with the pixel's tag, to the depth position 50 used for 3/2 display pixels, whidi 

also associated with that tag. In this way the element, "I '^''fi^/LP'''^^ f^'^ ^^''} S^?^^^ 

once its 'depth shape' at each pixel were defined, could ^+2, and a 50/50 mix of data pixels N and N+1 m 

be repositioned just by changing the one position depth f ^P^^^ ^ ^^l" interpolative ^heme is used to 

rather than by changing the entire table or, worse, the load 64(^pixel resolution data mto the 1536-pixel resoluUon 

depth (tag +value) stored at each pixel of the element. 55 ^"^^H g^°^P ""^.^^^l^ ^^^^^ [^'^^''''fP "^"^^ ^ 

Hie position could be a single value representing the apportioned mto 12 pixels (1 through 12) as follows: 

back, front or middle of the element. It may also consist of 

several numbers representing, for example, front and back, 

or position and thickness. In this way the element could be 2 "A 3 - o.4A + 0.6B 

both positioned and stretched in the depth direction by 60 4 - b 5 - 0.8B + 0.2c 6 = c 
changing only two values. 

Other variables associated with a tag could indicate how 

the table values were to be interpreted: as actual values, table 

indexes, or relative oflfeets (1, 2 and 3 above); positive, Similar interpolative schemes can be used to covert 

negative or signed; linear, l/finear, exponential or other 65 between other resolutions. It should also be noted that when 

non- linear function; etc. The interpretation of the values may data with relatively wide pixels is to be displayed in a frame 

also change as the position brings the element closer to or buffer of relatively narrow pixels, it may be parallax offset 
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by multiples of the aarrower display pixel, rather than the 
wider data pixel, to achieve finer depth effects. 

TTie reverse process can be used to derive interleaved or 
separate left and right images from the higher resolution 3-D 
display. For example three pixels of the 1536-pixel resolu- 
tion image would be combined and averaged to a single 
storage pixel of a 512-pixel resolution data file. 

Hie contradictory needs for high-resolution (many small 
pixels) to achieve small parallel offsets, and low-resolution 
(fewer larger pixels) for economy of storage and display, 
may both be addressed by a firame store where pixels of 
different sizes may be displayed at different parts of the same 
image at the same time. Such a frame store is described in 
applicant's co-pending U.S. patent application Ser. No. 
07/077,916 of which this application is a continuation-in- 
part. 

Sub-Pixel Positioning 

Information may be loaded into a display, either from a 
pre-existing image or from the 'brush' of a paint system; it 
may be loaded from information at the same resolution as 
displayed or (as described above) from a different resolution 
than displayed. In any of these cases the straight-forward 
approach leads to a minimum parallax offiset between planes 
of one pixel which may provide widely spaced planes or 
parts of objects in synthesized 3-D space. 

However, an intcrpolative technique, similar to that used 
to display information stored at different resolutions, may be 
used to provide sub-pixel positioning and, thereby, very fine 
and accurate parallax shifts which result in fine and accurate 
positioning in the depth direction. 

As an example here, consider display and data of the same 
resolution; an existing display line segment consisting of 9 
pixels (a) through (i); and a new data display segment 
consisting of 5 pixels (A) through (E). Table 1 shows how 
the new information is combined with the old information 
for five cases: case 1, with the new information obscuring 
pixels (c) through (g) of the old information but with no 
parallax offset (new information on the same plane as the 
old); cases 2 and 3, with additional left and right parallax 
shifts of one pixel; or, cases 4 and 5, with additional left and 
right shifts of Vioth pixel. Thus, parallax offsets, and thereby 
depth plane spacing, of Vio (or any fraction) of that normally 
achievable by display hardware of a particular resolution 
may be implemented by such a software technique. 
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However, two problems result from such sub-pixel posi- 
tioning techniques. First, it is not clear what depth should be 
stored with a pixel whose display contents consists of a 
combination of two pixels at different depths. Second, it is 
complicated, if not impossible, to correctly 'un -do', erase, or 
further process pixels at the edges of such objects whose 
display contents consists of combinations of more than one 
object. This leads to the next embodiment of the instant 
invention. 
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Maintenance of Separate Objects 

In order to avoid the above mentioned and similar 
problems, each separately painted plane, sub-plane, or other 
non-planer object, may be maintained as a separate object in 

5 separate buffers or sections of memory, other than the 
display's own memory. In this way, objects may then be 
properly parallax ofEset shifted, and left and right composite 
images built up in the portion of memory used by the 
display. With this scheme (as opposed to keeping only the 

10 composite image, only in display memory) objects may be 
moved in X, Yor Z without leaving ^holes' as objects reveal 
what is behind themselves. Additionally, when sub-pixel 
shifts, or other techniques such as 'anti-aliasing', result in 
display pixels that are composites of data pixels from more 

15 than one object, there is no problem due to not being unable 
to disentangle the composite information. Since the entire 
display image is reoonstmcted from the separate objects, the 
appropriate display pixel compositing is just re-computed 
from scratch. 

20 An additional advantage is that objects may be made 
transparent or painted transparently; or, because objects 
have depth associated with them, a light source(s) may be 
specified and correctly calculated shadows may be cast in 
3-D-space by some or all objects, to add realism to the final 

25 image. If either of these processes is done when just a 
composite image is stored, it is difScidt, if not impossible, to 
undo transparency or shadow casting. Shadow casting 
would probably be implemented as a post-painting process. 
However, if the objects are stored separately, as described 

30 above, not only can they be moved and re-oomposited, but 
they can be re -composited with transparency and shadows. 

Off-the-shelf high-speed image processing sub-systems 
such as the Intel DVI board for the IBM-PC are available. 
Such devices can achieve real-time or near-real-time com- 

35 positing of many separate and separately offset image ele- 
ments into a unified display image, or left and right pair of 
images. 

Alternately, instead of software compositing the final 
image in display memory, it is also possible that each object 
40 may be kept in its own display buffer, with the composite 
image created only on the fly by properly offsetting and 
mixing the output signals from the multiple display buffer 
sections. 

An additional advantage associated with this technique is 

45 that for many situations only a single copy of each object 
must be stored (with depth specification) and may be shifted 
in opposite directions prior to compositing for each of the 
left and right images. If alternate lines of an object/image are 
to be shifted in opposite directions for interleaved display, 

50 fine vertical detail may broken up into the two eye views, 
which may interfere with perception. Low-pass filtering in 
the vertical direction is one way to lessen such effects. 
Alternately, each source line may be used in each eye, 
although this will stretch the image vertically. 

55 Combining Separate Objects for Display 

In practice, most of the area of the left and right images 
composited from many objects will not be changing during 
the painting process; only those pixels under the brush or 
operational area will be effected. However, when an object 

60 is moved in any direction, much or all of the images will 
have to be rccomposited. To organize such operation, the 
human operator may be presented with two portions of an 
interactive program: one geared to painting or creating 
objects; and, another geared toward rearranging objects and 

65 fine-tuning the compositing process. 

Fine-tuning the compositing process may include various 
options as to how object boundaries are handled. Particu- 
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larly when using sub -pixel parallax shifts, pixels at the 
boundaries of objects may contain data from more than one 
object, with more than one depth. Visibly, anti-aliasing or 
mixing of the visible data from multiple objects, in a pro rata 
fashion, may be used at the edges. Alternately; the visible 
information from the object that is closest, or the object that 
covers most of the pixel may be used for the entire pixel. An 
additional problem is to determine what depth value is to be 
used and stored (for offset or obscuring purposes) with such 
composite pixels. Again, the depth value of a dominant pixel 
may be used, or the depths from several component pixels 
may be used. If a depth value is to be derived from a 
composite, simple or weighted averaging of the several 
component depth values may be used. 

Alternately, at these edges where the proper depth value 
is less certain, a technique may be adapted from applicants 
co-pending application Ser. No. 07/435,457, where color 
information was made less saturated or 'pure* as the edges 
of objects were approached. In this case the depth informa- 
tion will be made less 'pure' or * fuzzier' by adding a random 
variation or noise to those pixels at or near the boundary of 
two objects separated in depth. In general, the closer to the 
edge, the less pure the depth will be. In practice, this will 
amount to picldng depths for edge pixels that are randomly 
scattered between the near and far depths of: the component 
pixels, for anti-aUasing techniques or micro-positioning; or 
the bordering objects for other situations. 

A PREFERRED EMBODIMENT 

A* 3-D Computer Paint System* has been programmed in 
the computer language * C* by the applicant as a preferred 
embodiment of the instant invention. The program operates 
on the AT-VISTA display card installed in an IBM-PC/AT 
type computer. 

The display is configured for 1008 (nominally 1024, and 
may be increased to a nominal 1536) horizontal by 486 
vertical resolution, permitting a fine degree of horizontal 
parallax shifting (and, therefore, depth spacing) but config- 
ured for display on and recording by standard broadcast 
video equipment; as well as virtually all available electronic 
LCD-shuttered 3-D glasses. 

As explained more fully above, the 'row table* feature of 
the AT-VISTA can be reconfigured for five display modes: 
left/right in odd/even video fields, for 60 Hz operation; 
right/left in odd/even video fields, for 60 Hz operation; 
left/right in over/under full fi-ame, with a second blanking 
interval, for 120 Hz operation; right/left in over/under fuU 
frame, with a second blanking interval, for 120 Hz opera- 
tion; left image on top, right image on bottom, for glasses- 
less viewing of both images separately (a debugging mode). 

Facilities are provided for importing/exporting both 2-D 
and 3-D images, full screen or partial windows, at various 
resolutions (512, 640, 756/768, 1008A024, 1512/1536) with 
proper aspect ratio conversion. 

In order to simphfy programming and operation, both the 
'canvas* upon which the image is painted and the various 
brushes consist of continuous left and right parts; the inter- 
leaved display, when used, is only a function of the 'row 
table* feature. Each application of the brush, text or other 
function is thus carried out twice, once for each image, with 
the appropriate parallax offsets added to whatever normal 
2-D positioning data is obtained from the mouse, data tablet 
or keyboard. 

This paint system is suppUed with an unusually large 
number of user adjustable parameters and options, and 
virtually all such options may be separately adjusted for the 
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left and right canvases. Both field and frame cursor 
movement, as described above, are available. 

At each pixel, eight bits each are stored for Red, Green 
and Blue visual information. The fourth eight-bit channel is 

5 used as follows. Seven bits are used for 'Z' or depth 
information. Of the 128 values thus possible, 101 are used: 
50 into the screen; 50 out of the screen, into room space; and, 
the plane of the screen (no parallax offset). If the paint 
system is left with the plane of the screen selected, it may 

10 thus be used as a standard — but, very flexible — 2-D paint 
system. The eighth bit of the fourth channel is reserved as a 
protection bit, whose use will be described below. The Red, 
Green, Blue, Depth and Protection data may each separately 
(and separately for each of the left and right canvases) be 

15 selected for writing to the frame buffers memory, or masked 
from writing to memory. 

In this embodiment, where input comes from a 2-D data 
tablet, the user selects a single plane at a time upon which 
painting is to occur, or to which a flat or multi-depth function 

20 is to be referenced. Alternately, when connected to a 3-D 
input device, the selected plane may also be dynamically and 
continuously selected along with the X and Y position. 
3-D Menu Elements 

The 3-D paint system has many usual elements, menus of 

25 buttons, color and brush palettes, sub -menus, etc. but these 
are displayed and work in a uniquely 3-D way. 

All buttons and palettes float upon a menu plane which, 
in turn, floats above the canvas; this effect is achieved by 
relative parallax offiset. Similarly the text of the button floats 

30 above the plane of the, button. In addition, the buttons cast 
drop shadows in 3-D. Normally drop shadows are dark areas 
the same shape as the casting object offset down and right 
(or up, or left) and obscured by the casting object. In this 
case, in addition to the drop shadow offiset, parallax offeets 

35 are added to the left and right shadows to make them appear 
behind the casting object. Similarly, elements cast shadows 
onto palettes, palettes cast shadows onto the menu plane, etc. 
Whenever selected or *hit% buttons and other elements 
animate by depressing and returning as appropriate by 

40 altering the parallax shift of the selected element, through a 
number of positions, over several frame -times. 

For certain functions windows open into the menu, and 
other user interface elements or messages are revealed. One 
such window is the * ruler' a device with a set of tics for each 

45 plane that indicate the depth value of the selected plane (with 
progressively longer tics at intervals of 5, 10 and 50), 
whether that plane has been previously selected for painting, 
or whether that plane is currently set as 'protected' or 
masked from painting. 

50 It should be noted that the Apple Macintosh computer, the 
IBM Windows operating system and other systems provide 
so-called Graphic User's Interfaces (GUIs) which incorpo- 
rate menus, buttons and stacked windows on a 'desktop'. 
The desktop often gets crowded and confusing as many 2-D 

55 elements partially obscure each other. It is potentially useful 
to use the stereoscopic display techniques described herein 
to make such GUIs into 3-D user interfaces to assist users 
navigating through such crowded interfaces. 
All options and features may be selected for the right 

60 channel, left channel, or both; and may be copied right-to - 
left and left- to-right. Different colors and brush halves are 
displayed for each eye as appropriate. 

The cursor is positioned in Z as well as X and Y by the 
use of parallax oflkets. In addition, flat rectangular cursors 

65 used to delineate areas for processing, storing, editing, etc. 
have been generalized to 'bricks' which delineate a rectan- 
gular solid space in the 3-D canvass. 
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Protection and Other Restrictions on Painting Pixels Brush List 

Various methods are available to protect particular pixels In this paint system brushes may be any shape within a 

from being effected by the painting process. Each of these 16x16 pixel rectangle. However, Uke the canvas itself, each 

various types may be selectively enabled or disabled for bmsh is 'disinterleaved into a right and left half, each only 

each of the two images. 5 8 pixels high by 16 pixels wide (128 pixels in all, for each 

As stated above, the eighth bit of the depth channel is halQ. The bmsh itself may have a single bit, or 8, 24 or 32 

reserved as a pixel-by-pixel protection bit. If this option at bits of information stored at each pixel. One may chose to 

painting is selected during a painting function, the protection use the two halves of the brush as a left/right pair or have 

bit is turned on for each pixel changed. A complementary either half repeated for both halves of the canvas. Each brush 

function prevents such protection tagged pixels from being lo half may also be independently turned on or off for painting, 

painted over. Hie paint system has many different painting modes, 

By selecting both the turning on of the protection bit by some of which require complex calculation at each pixel 

painting, and the use of the protection bit to prevent painting effected. Therefore, in order to keep the operation of the 

over, at the same time, a very useful function results. If one program running as fast as possible, unnecessary calcula- 

has selected a partial tinting paint function (e.g., tint all 15 tions must be avoided. 

pixels hit by the brush 10% green) then repeatedly going Part of painting is determining which pixels are excluded 

over the same pixel will tint it more and more heavily. This by the shape of the brush within its rectangle, by the various 

may in fact be what is desired. But, if it is desired to protection schemes, by the restriction of depth values, 

uniformly tint an area to the same degree, each pixel should restriction of color values, or by the requirements of the 

be effected only once; this is very hard to do free-hand. This 20 particular painting mode (for example, luminance paint only 

technique guarantees that each pixel is effected no more than paints if the pixel to be painted is brighter than the pixel 

once. Using more bits for a 'protection plane' provide the contents already at that position). Two schemes arc 

abiUty to count the number of times a pixel has been painted described below. 

or *hit' and thus more functional flexibility. For example, First-, an essentially one-bit version of the brush may be 

such a protection counter could be used by the program to 25 kept, which operates as a mask or picture of which pixels are 

permit a pixel to be tinted heavily on its first hit, and to be effected. At the beginning of each application of the 

progressively less heavily tinted on subsequent hits. Or to brush, the various protection schemes are checked and bits 

only perform a function after some threshold of hits has been in the mask are turned off if protection will ehminate that 

reached. pixel from being effected. Only those pixels still represented 

By turning on the protection bit when painting, but 30 in the reduced brush mask are then appUed to the canvas, 

masking the writing of Red, Green, Blue and Depth values. The problem with this scheme is that the various protection 

the operator may paint an area of 'protection from further schemes and many elements of the paint amount to a 

painting* (Fixing) without effecting the contents on the decision being carried out (is the protection bit on, is the 

screen or depth values. Areas may be similarly ^imfixed' by depth value separately protected, etc.). Wtiile finding that a 

painting with the protection bit turned off and, again, mask- 35 bit is turned off in the one -bit image of the brush eliminates 

ing Red, Green, Blue and Depth. the need to proceed further with a particular test, the test of 

Similarly, if Red, Green and Blue are masked off, but whether each of the 128 pixels (for each bmsh half) are on 

Depth is not, an 'invisible sheet of glass' at a particular depth or off in the brush image must be repeatedly tested for each 

may be created which will protect everything behind it from pass of protection or final painting as the brush is laid down 

being painted by a brush behind (further than) the plane of 40 once. One may test 128 pixels for each of M tests or one may 

the 'glass' but which may be painted over by a brush closer perform M test for each of 128 pixels, the result is stiU 128 

than the plane of the glass. times M tests. 

Other types of protection are provided. Each plane may be To gel aroujid this problem the instant systems instead 

explicitly protected by selecting it as such on the ruler. keeps each brush as a list of pixels. Each list of bytes starts 

Facilities are provided to turn on or off specific values or 45 with a counter that teUs how many entries are in the Ust: 

ranges and complex combinations may be built up this way. between 0 (no pixels in the brush) and 128 (all pixels on). 

Then pixels are painted, or not, based on their depth value Then the list follows with that many entries listing the 

rather than on the state of their own protection bit (although location (in raster order) of each pixel in the brush. In raster 

both methods may be used simultaneously). Pixels may also order, from left to right, top to bottom, the upper left comer 

be protected because their display (Red, Green, Blue) con- 50 is 0, the lower right comer 127. Since a rectangle of image 

tents are a specific value or in a specific range of color is brought into or taken from the computer's memory by the 

values. AT- VISTA display as a linear array in raster order, this 

Also, options are provided to restrict application of the scheme works well, 

painting function to: all depth values, one depth value, a Thus, if the brush contains very few pixels, the list is short 

range of depth values, all but one depth value, all but a range 55 and very few pixels must be considered. No time is wasted 

of depth values. repeatedly considering and then rejecting as not vaMd the 

LasUy, when painting at a particular plane, it may be many 'off' pixels in a brush image. As each protection 

separately specified whether painting will or will not effect sdieme is applied, rejected pixels are removed from the list 

pixels that are: closer, further, or equal to the plane being since, once rejected for one reason, it is no longer necessary 

painted. The usual mode is that pixels that are closer will not 60 to determine if a pixel is to be rejected for a second reason 

be affected. as well. 

These last two restrictions based on depth may be used in The list is shortened by taking the last entry, placing it in 

tandem. But, for some types of functions the first is applied the rejected pixels position, and decrementing the first list 

prior to calculation (cutting the points considered for the entry (the counter) by one. In this way, whether a pixel is 

calculation of a digital filter, over a 3-D kernel, for example), 65 'on' or 'off' never needs to be considered; off pixels are 

and the second after calculation (culling which points are to never on the list for consideration to begin with. The list can 

be updated on the screen). * only get shorter as pixels are rejected by various tests. 
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PaiDting Modes 

The 3-D paint system has many different paint modes; 
some are similar to 2-D paint systems; others are unique to 
this program. Standard painting may be achieved by using a 
one-bit * rubber stamp-like^ brush with a color chosen from 
the palette. Alternately a brush may be a 'picture^ with Red, 
Green and Blue (and even Transparency or Depth) values 
stored at each pixel. This bush may than be appUed to the 
canvas in a straightforward manner, or my be used to tint 
(diange hue but not luminance) or shade (change luminance 
but not hue) the canvas. The entire result may also be 
modulated by a transparency factor which keeps some 
proportion of the old pixel's display value. 

In addition to the other protection schemes described 
above, several versions of * luminance* paint arc available 
which limit painting to pixels whose new value will be 
above (or below) a threshold based on an average of the Red, 
Green and Blue values; a television luminance calculation; 
or separate comparisons for each of the Red, Green and Blue 
plane values. 

Other types of painting perform various mathematical 
calculations (e.g., logical AND, OR, XOR, and their 
complements, addition, subtraction, multiplication (with and 
without clipping), etc.) on the current and brush pixels data 
and use the results to replace the old pixel (again in regular, 
tint, shade or transparent fashion). 

Still other types of painting perform a calculation based 
only on the canvas pixels' values, based on some parameters. 
For example, several image processing functions can be 
attached to the tip of the brush to selectively adjust level, 
gain, contrast, saturation,. color correction, etc. on a free- 
hand basis. 

Other types look to a neighborhood around the affected 
pixel and perform various digital filters that may blur, soften, 
sharpen, etc. the image; or, may smear pixel values in the 
direction of the motion of the brush, 

'Air' brushing is a technique that lays down a transparent, 
soft-edged swipe of color in two dimensions on the canvas. 
There is also 'Spray' painting that does so but with a thick 
curved tube (by use of depth values and parallax offset of 
individual pixels) several planes thick; it has a dense center 
with a *haze' around it. Asimilar tube-like mode with a 'hot* 
center and softer *glow'' is dubbed *neon' paint. In these 
* thick' painting modes the depth values may be updated, 
creating a substantial object. Alternately, depth values may 
be left unchanged so that no record of the spray's existence 
would be made, it would be * insubstantial'. Additionally, the 
core may be made substantial with the *glow' or 'haze' made 
insubstantial. 

In addition to arbitrary 16 by 16 pixel brushes. Larger 
computational brushes are available. A 'disk' is round with 
imfform density and a sharp but anti-aliased edge; a 'cone' 
is round with a dense center which falls off to transparent in 
a controlled fashion; a 'ball' is like a cone but, again, with 
depth values that give it thickness and roundness in the Z 
direction. Storing transparency values for individual brush 
pixels is used to implement the soft edges or glows described 
above. 

One last painting mode wiU be described; Shape paint 
painting is done on the surface of a 3-D shape already in the 
frame buffer, changing the surface look but not the depth 
shape. A color and brush are picked as usual. However, 
instead of picking a depth and painting at a particular 
parallax offset, when Shape paint mode is picked, pairs of 
pixels with the appropriate cursor Y value are compared. 
When two pixels (in the two images) arc found whose stored 
depth values are the same and their relative parallax offset 
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consistent with that depth value as applied to the non- 
parallax- adjusted X value of the cursor, that pixel pair has 
the right X, Y and 'virtual' Z to be painted. 

'Solid' objects may be built up from contours at different 

5 depth planes (hke a contour map) or may be rendered by 
other software and imported, or rendered by the 3-D paint 
system. In addition, Ubraries of precomputed shapes may be 
made available to the operator, much like 2-D clip -art. In any 
of those cases, shape paint will permit the surface of those 
objects to be re -painted without disturbing their shape. 

The ability to create 3-D objects by slacking painted 
contours, as well as to import and combine 3-D objects from 
other sources, and to 'Shape paint' provide a flexible mecha- 
nism for a painter or illustrator to create 3-D objects and 
environments. One particularly useful appLcation for this 

15 system will be the creation of such objects and environments 
for inclusion in Virtual Reality simulation systems. These 
'illustrations with depth' can be converted by software 
utilities to the appropriate file formats for such VR rendering 
systems. 

20 Lines and Other Primitives 

Although the depth planes are noticeably separated, lines 
and other primitives (rectangles, polygons, etc.) may be 
drawn to smoothly span the distance between planes. A 3-D 
line or polygon may be drawn by adding appropriate par- 

25 allax offsets to the end-points or vertices of the object 
(different for the left and right image) and using existing or 
weU known algorithms for drawing the adjusted image 
twice: once for the left, once for the right. 

A problem results in that while the image may appear to 
be smoothly spanning a 3-D space, the depth values stored 
for those visible pixels will be quantized or *stairstepped'. 
This situation may be kept in mind as a limitation of the 
system or, alternately, the depth values may be 'dithered'. 
Dithering randomly selects one value from the closest 
available quantized depth values above Xand below the 
desired depth value. The probability of selecting one or the 
other is adjusted depending upon how close each available 
value is to the desired value. ^ 
In this way a ' fuzzy' rather than stairstepped depth image 

^ is created and when further painting or other image func- 
tions interact with the object, a fuzzy but smooth looking 
intersection with the new object results. 

If 16-bits are available for depth buffering, the entire issue 
becomes moot because enough resolution in the depth 
direction is available for smooth depth specification wdthout 
dithering. 

Additional Features, Techniques and Embodiments 
Serial Parallax 

It must be understood that the parallax shifts associated 
50 with the left and right eye views, in effect, change the point 
of view (POV) of the viewer of the multi-plane image. By 
adding coordinated, rather than opposite, shifts, the POV of 
the two-eyed viewer may be shifted. Several useful varia- 
dons result. 

55 First, even without glasses or any kind of binocular 
stereoscopy, the POV may be shifted in 'wiggling' motion to 
produce 'motion (serial) parallax' which can convey much 
depth information. Although the perceived wiggling of the 
image can be annoying, and therefore is of limited user for 

60 long entertaimnent presentations. For short term 'bringing to 
life' of stacked multi-plane images, such as those created by 
the instant system, this method can be very useful to convey 
3-D information without any glasses. For, scientific, 
industrial, military, educational or medical imaging, 3-D 

65 visuahzation of data can be achieved without glasses. 

The appUcant has found that five such images — with no 
parallax oflket (N), slight ofeets to the right and left (r and 
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I), and greater ofiEsets to the right and left (R and L) — a television signal and by the use of a frame store, show each 

strong 3-D effect can be achieved by alternating them in the twice. Such a framcstore device may be easily modified to 

following order: L, 1, N, r, R, r, N, 1, L, 1, N, r, R, r, N, 1, etc. solve the flicker problem associated with 3-D electronic 

As explained briefly above, the appUcant has used the LCD-shuttered glasses which dehver only 30 fields to each 

Intel DVI board to composite images in real- or near-real- 5 eye. Similarly, because the odd lines go the one eye, and the 

time utilizing the multi-plane techniques described herein. even hnes to the other, vertical resolution to eadi eye is 

Such a system can be programmed to update the POV, or halved. A television with a frame store device would be able 

parallax shifts between planes, to implement the above. ^o fiU in those gaps during display ApphcaUon of the 

Additionally, when coupled to input from a head-position- fPP^^^^J ^ ^^^^1^ ^ Spatial-Temporal Si^ature (STS) to 

. I • J • nrU/ t- J » J 4- 1 * « this technique will prove useful. See apphcant s Ser. No. 

tracking device, the POV can be updated continuously to lO n-nnm ni< 

. J- 1 r * . *i_ 07/077,916. 

create a display from the appropnate perspective as the Additionally, a new type of low-cost electronic stereo 

viewer moves around the room; this is a very powerful 3-D ^^^^^ ^^^^ ^^^^^ ^^^^ lights (or LEDs) are 

(though not bmocular) effect. The combination of 3-D flickered, alternating with each eye in synchrony to the 

glasses and the head-tracker (or a head-mounted display) alternating left/right pattern. Hie hght for one eye is on 

produce an even more powerful effect. In that case, not only 15 ^jjile the image for the other eye is displayed. Although not 

head position but orientation may be captured and each eye bright enough to completely disturb vision, the small flash in 

may be displayed with proper horizontal and vertical par- one eye will suppress the perception of the unwanted image 

aUax ofiEsets, since head orientation may tilt the eyes. in that eye. The Ught may be directly flashed into the visual 

Such a system is a low-end alternative to costly full- field or, preferably, through a diffusion element. Alternately, 

immersion hehnet-based Virtual Reahty displays. When 20 haff- reflective lenses may be used in these glasses which 

using such a system and drastically changing the viewer's will reflect the internal light flash but otherwise pass the 

POV (particularly when the multi -plane image is derived television image to the eye. There wiU also have to be an 

from a 2-D image or film/video sequence as described in opaque barrier to prevent cross-talk between the two hghts 

applicant's U.S. Pat. No. 4,925,294) holes in the image may and two eyes. 

develop, and the reverse sides of objects may need to be 25 Interleaving Other than by Field or Frame 

displayed. It is just such as system as described herein that Generally, left and right image pairs are alternated as 

will be used to fill those holes or 'paint' the reverse sides of video fields or frames. However, interleaving of smaller 

objects so that not only will 2-D images and motion image sections can be used to advantage for improved 

sequences be able to be adapted to 3-D stereoscopic perception. Individual lines or sub-lines can be alternated 

displays, but may also be adapted for use with Virtual 30 from the left and right images. Faster flickering electronic 

Reality systems. LCD-glasses, than currently available on a low-cost basis, 

Stereo Broadcasting would be required to decode such an image. However, if 

Systems have been disclosed, such as that by NYIT, using passive red/green anaglyphic glasses, flickering is not 

where two images may be combined in such a way that by an issue. 

proper frequency filtering either one may be displayed. This 35 The problem with using red/green anaglyphic images with 

has been disclosed as intended to be utilized to transmit two television is that the color encoding process lowers the 

entirely independent image streams. However, it is also resolution of the color portion of the signal. However, if 

possible to transmit the left image as one image stream and whole individual Hnes (or large line parts) of the image were 

the right image as the other stream. Alternately, the single uniformly tinted red (for left) or green (for right) these large 

center image stream may be sent and a second, perhaps 40 sections would be displayed at high resolution because only 

lower-quality, image stream (by the above or any other their luminance portion would be changing, not the color 

method of hidden information) may contain depth informa- portion of the signal. Thus, high-resolution anaglyphic 

tion only. Then, as described above, the depth image may be images could be broadcast or taken from videotape, 

used to distort the single image, once to the left, once lo the For either of the techniques described above, the way that 

right, to create stereoscopic paired image streams. 45 the left/right information is distributed over the frame may 

When distorting such a single image into left and right be changed from frame to frame in a non-trivial manner, as 

images, (or as described elsewhere in the instant apphcation) a way to improve perception. This technique, called a 

holes may develop as shown below with B to the rear of A variable STS, is described in some detail in appUcant's 

and C. co-pending application Ser. No. 07/077,916. 
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Such holes (**) may be automatically covered by stretch- 
ing the available material at either end, or by interpolating 
the material on either side of the holes. In one approach the 
holes may be fiUed with values derived from interpolating 
from A to C. Alternately, the hole in the right image may be 
covered with material interpolated between A and B; and, 
the hole in the left image may be covered with material 
interpolated between B and C. 

Also, frame -rate-doubling televisions are now commcr- 
ciaUy available. These take the 60 fields of a standard 



3-D Mouse 

As described above, the 3-D paint system will operate at 
a continuously and dynamicafly selected depth with the 
addition of a 3-D input device. Such a device is described in 
the following pages. 

For inputting positional information to digital computers, 
mice (basically upside-down trackballs) operate with a 
single roller-ball or optical sensor and give two axis inputs: 
X (left-to-right) and Y (away/toward the user). One, two or 
more buttons allow for selection type input as well. Flat X-Y 
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data tablets provide similar fuQctionality, with one on/off 
'button' function often tied to the pen tip. 

For many ' flat screen oriented' applications this is 
adequate. However, for iDther applications additional input 
axes are advantageous. Tablet and pen units exist where tip 5 
pressure (rather than just on/ofif contact) is measured and can 
be used for a third axis. However, such a solution is costly, 
delicate and takes up a large amount of desk space. Further, 
for many, especially for 'non-computer-artists' who are the 
vast majority of users, a mouse type device is preferable to lO 
a tablet and pen. 

The present discussion relates to a class a mouse -like 
input devices that employ multiple roller-balls (or optical 
sensors) to provide additional input axes and a versatile user 
interface. 15 

If a second ball is added to the underside of a mouse, input 
on a third axis is possible — a twist about the Z axis; i.e., the 
axis perpendicular to the two axes of normal mouse move- 
ment. Such input could be used as a rotational parameter, for 
example, to point a screen cursor in a particular direction, 20 
not just position the cursor on the X-Y plane; or, for other 
rotational information, such as to rotate 3-D objects in a 
computer graphic or CAD system. Alternately, the input may 
be used as s magnitude parameter, for example to translate 
along the Z-axis, e.g., clockwise (CW) twist for up (with 25 
respect to the desktop, or out with respect to the screen) and 
counterclockwise (CCW) twist for down (or in). Or, it may 
be used for sign or selection information for a ^double 
button' like a two way switch or knob, e.g., CW twist for 
'plus' or select, CCW twist for 'minus' or cancel. 30 

The two-ball mouse comprises two trackballs mounted 
fore and aft to (of) the center point of the Y-axis. 
(Alternately, the two balls could be mounted to the left and 
right of the center of the X axis; but, this wide mouse would 
probably be harder to hold.) For normal X (side -to-side) and 35 
Y (up/dowa) operation, the two balls operate in tandem, 
giving (theoretically, and at least approximately) equal out- 
put. For a twist, CW as seen from above, the rear ball will 
register a positive or rightward X output, the rear ball a 
Degalive or leftward X output. A CCW twist will produce 40 
complementary output. 

The output of both balls can be sent od to the host 
computer for integration there, or ihey may be integrated by 
the circuit in the two-ball mouse for transmission of X, Y 
and Z data (along with the button hit information as well). 45 

The two -ball mouse's software driver will pass on the X, 
Y, Z and button-stale information to applications pro- 
grammed to properly interpret it. Alternately, in a standard 
mouse compatibility mode, only the standard X, Y and 
button information would be presented to the application, so 

In mouse-like manner, the two-baU mouse may be Hfted, 
unwound and replaced on the pad for continued twisting. 
However, an alternate interpretation of the input by the 
software, either in the mouse or in the host computer device 
driver, would allow for a 'wind-up mode'. Like a mechani- 55 
cal watch stem, the CW (or CCW, perhaps chosen by 
whichever move started the sequence) twists would 
accumulate, the CCW would be ignored. Alternately, the 
CW and CCW twists could be summed for a magnitude, the 
sign being specified, e.g., by a 'button' or by which twist 60 
direction started the sequence. Which twist mode is in 
operation may be specified by the user via a button or a menu 
selection, or may be determined by the appUcation depend- 
ing on which software function were in operation. In 
particular, software may allow the user to turn off wind-up 65 
mode by making a normal X or Y axis motion; pausing 
longer than some threshold between twists; making repeated 
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turns in the same direction, or further than the normal 
back-and-forth wind-up twists; or by some other pro- 
grammed gesture. 

The advantage of such a wind-up mode is that it allows 
the mouse (and more importantly the user's wrist) to twist 
back and forth, without lifting the mouse for the imwanted 
directional twists, and to accumulate a total rotation much 
further than the range of the human wrist would allow. 

A similar mode could also be implemented to avoid 
having to go through the ' drag-lift-drag-hft-drag' sequence 
for long X and Y motions. However, due to ergonomics, this 
feature is much more important for the much more restricted 
rotational movements. 

In the twist mode described above, the axis of twist is 
through a point between the two balls, and the front and rear 
balls act in opposition. In an alternate 'pin and wag' mode, 
the axis of twist would pass through the center of either the 
front or rear trackball. In such a mode the pinned ball would 
see (theoretically, or at least approximately) no movement 
and the other ball would 'wag' left and right, along the X 
axis with a slight curve, for balls mounted front and rear. So, 
in addition to twist CW/CCW, one could pin the head ball 
and 'wag the tail' back and fourth, or pin the tail and wag the 
head back and fourth. 

As a way to implement a variation on the 'pin and wag' 
mode, a 'rocker' pin or ball would be placed between the 
two trackballs. This pivot would extend somewhat beyond 
the two trackballs and would allow the user to pivot to have 
either the front or rear ball contact the mouse pad, without 
the other making contact. In that way, the contacted ball 
could be 'wagged' around the pivot, in a manner similar to 
above but with a smaller pivot radius. But, for even more 
flexibility the pivot point would have a small roller-ball in its 
tip, much like a large-bore, inkless, ballpoint pen refill. As 
such, the rear (or front) ball and pivot could be moved left 
and right along the X axis, much like the wagging mode; or, 
back and fotirth along the Y axis. 

A spring load in the pivot point would allow the user to 
apply a small amoimt of additional pressure to have both the 
front and rear balls contact the mouse pad at the same time. 
In that case, the dual-ball X and Y motions, as well as the 
twist mode, would be available. 

Thus, there would be a total of at least seven axes with the 
dual trackball mouse, as opposed to only two (X and Y) with 
a standard mouse. These are: X and Y pivoted for the front 
only, X and Y pivoted for rear only, X and Y with both balls 
contacting and moving in a coordinated way and, lastly, 
twisting with both balls contacting in an anti-coordinated 
way. 

In addition to increasing the number of input axes from 2 
to 7, a dual trackbafl mouse would permit the new gestures 
to be added to the X-Y positioning and button clicking of the 
standard mouse. As on example, a 'wrist flick' could be used 
to implement a number of useful functions such as a 'page 
flip' to browse through a text file as if it were a book. 

A flick consisting of a short CW motion followed by a 
return CCW motion would constitute a jump ahead one page 
command; a CCW-CW combination, a turn back one page. 
A(a adjustable) threshold of, say, about 5 degrees would 
prevent accidental page turning from minor jiggUng of the 
mouse. A larger flip would turn 2, 5 or 10 pages (again, 
perhaps user selectable) and a very large flip would turn 50 
or 100 pages. The degree of 'largeness* would be increased 
by: increasing the CW/CCW rotation to 10, 15 or more 
degrees; increasing the speed of the flip; increasing the time 
between the CW and CCW phases of the flip, or between 
two successive flips; or, to make a nimiber of flips in quick 
succession with 1, 2 or 3 flips indicating 1, 10 or 100 pages, 
for example. 
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Many other gestures and axis combinations can be devel- Thus, if (instead of North, South, East and West) the NE, 

oped with such a device and the proper software. For NW, SE, and SW directions were used, four stable configu- 

example, in a computer painting or drawing program, cal- rations would be achieved. Although each of the four comer 

Ugraphic writing may be implemented by drawing in X and balls would be used twice, each of the pairs is unique and, 

Y as usual and, in addition, controlling the angle or tilt of the 5 therefore, no confusion would exist. An added benefit of this 

*pen' by twisting the mouse. Similarly, for 3-D modeling or embodiment is that, since two comer balls are used in each 

sculpting, an oriented chisel-like function may be implc- mode, X,Y, and twist axes are accessible for each of the NE, 

mented. NW, SE and SW comers, for a total of twelve axes without 

Or, using the two trackballs independently, with the having an active center roller-ball; 14 axes with an active 

rocker, two separate objects may be controlled simulta- lo center. 

neously without having to switch back and forth between In practice, the entire design may be rotated 45 degrees, 

functions, using a button click. By pressing harder, and so that the active comer roller-balls would be located at the 

having both balls contact the mouse pad, the two objects NE, NW, SE and SW corners, with stable tilt configurations 

may also be moved together in a coordinated manner. at N, S, E and W. 

More than two roller-balls may be used for additional 15 Of course, with a dexterous user, all eight directions (N, 

input axes and increased system flexibility. Many configu- NE, E, SE, S, SW, W, NW) may be used for a total of 20 axes 

rations arc possible and within the scope of the present without an active central ball; 26 with, 

invention but, in particular, four- and five-ball configurations In addition to the specific configurations, uses and ges- 

wiU be discussed blow. tures described above, with additional input axes available, 

Consider a situation where four balls are placed at the four 20 the possibUities for games, 3-D animation, computer aided 

compass points (East, equal 0 degrees, is to the direct right). design, machine or robot control, or other user interfaces are 

All balls would act in unison and would thus provide only greatly increased. 

X, Y and twist axes; no improvement over two roller-balls. Finally, all of the principles and techniques described 

However, if an extended, rolUng pivot were provided in above can also be applied to multiple transducers on other 

between the four roller-baUs, each of the four balls could be 25 computer input devices. For example, two (or more) pen/ 

made to contact the mouse pad independently, by tilting the stylus or puck/cursor receptors may be put in the same 

whole mouse to the right, left, away or toward the user. Each housing and queried (in parallel or serially) to generate not 

of the four balls would provide input along the X and Y axes, only position information but orientation information as 

for a total of eight axes. well; or, two trackballs may be mounted in such a way as to 

However, now consider the situation where the central 30 be covered by the palm and thus input direction information, 

pivot ball were also an active input roller-ball. When used not just X and Y positional information , 

with each of the four corner baUs, the central ball would It will thus be seen that the objects set forth above, among 

provide a pair of balls and this be able to provide a twist axis those made apparent from the preceding description, are 

as well as X and Y, for a total of 12 axes. Additionally eflSciently attained and certain changes may be made in 

(perhaps, depending upon the amount that the central ball is 35 carrying out the above method and in the constmction set 

extended and the dexterity of the user) by balancing forth. For example, the substitution or combination of van- 

carefuUy, so that none of the comer balls touched the pad, ous technologies, including but not limited to: analog and 

the central ball alone could provide an additional set of X digital circuitry; special purpose circuitry and general pur- 

and Y axes, for a total of 14. pose circuitry running software; various 3D displays, such 

Such an embodiment may be particularly useful for 40 as those employing passive or active glasses, lenticular 

systems used to design and produce 3-D computer anima- screens, CRT displays, holographic film, virtual reahty and 

tion. In such systems it is often required to specify (at least serial motion parallax; trackballs, mice, styluses and data 

for key frames) the X, Y and Z positions as well as rotations tablets, or other position or mput sensors in coordinated use 

around the X, Y and Z axes, for both objects and the as described herein; and, various ways of storing image and 

theoretical ' camera' . These four groups of 3 parameters may 45 depth information. Accordingly, it is intended that all matter 

each be assigned to the four comer roUer-baUs, each with 3 contained in the above .description or shown in the accom- 

axes. Thus, without having to switch between modes, all of panying figures shall be interpreted as illustrative and not in 

the required positioning information may be controlled a hmiting sense, 

gesturally. The same corner roller-balls may be used to enter Now that the invention has been described, what is 

other information in other modes; mode-switching would be 50 claimed as new and desired to be secured by Letters Patent 

achieved with the active center ball, or otherwise. is; 

It should be noted that for optical mice, (where an optical 1. An improved method for creating image information by 

sensor coxmts a number of lines passed over on the mouse using a computer paint system, wherein painting functions 

pad, instead of a roller ball physically rolling over the pad) include at least some function available in standard com- 

a small contact switch would be mounted for each sensor to 55 puler paint systems including brush -oriented painting and 

sense which of the four 'balls' were being tilted toward for pen-oriented drawing functions, wherein said improvement 

selection. Or, some other set of orientation sensing devices, comprises the creation of three-dimensional image 

for example, mercury switches, would be used. information, wherein said three-dimensional creation is 

An additional variation provides extra stability to the effected by maintenance of a plurality of image information 

embodiment described above. Consider the situation where 60 memories and application of at least one paint function to at 

the East or right roller-ball is chosen, by pivoting the entire least two of said image information memories and wherein 

assembly, on the central ball, to the right. In that orientation, the application of said at least one paint function in a first of 

the device is not entirely stable; it may, additionally, be tilted said image information memories is offset with respect to the 

North or South, or balanced in between. When tilted North application of the same paint function in a second of said 

and East both balls touch the pad (as well as the central ball) 65 image information memories. 

and a stable configuration is achieved; similarly, for South 2. A method for integrating 2D and 3D image elements by 

and East. utilizing the method of claim 1. 
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3. A method for modifying a 3D image by utilizing the 16. A method as in daim 1, wherein said plurality of 
method of claim 1. image information memories is two. 

4. A method as in claim 1 wherein said offeet may be 17. A method as in daim 1, wherein said plurality of 
adjusted by a fraction of a pixel for at least one image image information memories is more than two. 
information value by application of interpolation to appor- 5 18. A method as in claim 1 comprising in addition the step 
tion said image information value to more than one image of conveying at least some of said image information 
information memory location. embodied as a video signal via an information bearing 

5. A method as in claim 1 wherein a painting function is medium. 

truncated in such a manner that for progressively larger 1 9. A method as in claim 1 comprising in addition the step 

ofifeets an extreme edge of the area to which said painting lO of conveying said image information via a left and right pair 

function may be applied is progressively restricted in order of film images. 

to avoid the apphcation of said painting function to an image 20. An improved method for creating final image infor- 

intended to be displayed to one eye without it also being matioo by using a computer image creation/modification 

applied to an image intended to be displayed to the other eye. system, wherein image creation/modification functions 

6. A method as in claim 1 wherein said plurahly of image 15 include at least one function available in standard computer 
information memories are each uniquely associated with at paint systems including brush-oriented painting functions, 
least one of a plurality of image information elements, and wherein said improvement comprises storage of image depth 
left and right images are created by applying parallax ofibets information in addition to image color information, and 
to at least some of said plurality of image information wherein said image depth information specifies other than a 
elements, and wherein said parallax offsets are determined 20 single plane perpendicular to the depth axis when depth 
as a fimction of depth information associated with at least excursions representing surface texture are eliminated, and 
some of said image information elements. wherein said at least one function produces dynamic visible 

7. A method as in claim 6 wherein said offsets may be feedback in the display of said final image information of its 
adjusted by a fraction of a pixel for at least one image effect upon at least some of said image information during 
information memory value by application of interpolation to 25 its operation. 

apportion said image information value to more than one 21. A method as in claim 20 wherein obscuring of one 

image information display pixel. image infonnation element by another is based on said 

8. A method as in claim 1 wherein said plurality of image image depth information. 

information memories comprises two, a left image infonna- 22. A method as in claim 20 wherein an additional 

tion memory and a right image information memory, and 30 improvement comprises at least one function to paint with a 

said ofiset is binocular parallax oflfeet. brush that applies both color value to a color image memory 

9. A method as in claim 8 wherein said left and right plane and depth value to a depth image memory plane, 
image information memories are each maintained as sepa- 23. A method as in claim 20 wherein said depth informa- 
rate contiguous areas of memory and combination of the tion is sub-divided into depth tag and depth value informa- 
contents of said left and right image memories is effected 35 tion and a depth specification to be used is derived as a 
only upon display. function of the depth tag and depth value. 

10. A method as in claim 8 wherein said left and right 24. A method as in claim 20 wherein an additional 
image information memories are arranged in an interleaved improvement comprises that for at least one image process- 
fashion within a single information memory. ing function a set of pixels input to said image processing 

11. A method as in claim 8 wherein an image, element is 40 function and a set of pixels effected by output of said image 
loaded into both left and right image information memories processing function are dependent upon depth information 
without offset, the contents of said image information associated with at least some of said input and effected 
memories are then parallax shifted with respect to each pixels. 

other, and subsequently an additional image element is 25. A method as in claim 24 comprising in addition the 

loaded into both left and right image information memories 45 step of conveying at least some of said image information 

without ofket. embodied as a video signal carried by an information 

12. A method for displaying information with the percep- bearing medium. 

tion of depth comprising the synthesis of a plurality of 26, A method as in claim 20 wherein an additional 

images each created by the process of claim 1, each of said improvement comprises inclusion in said created image 

images constructed from a different viewing position, and 50 information of shadows cast by at least one first element 

displaying said plurality of images in an oscillatory sequen- onto at least one second element whereby a shadow casting 

tial manner. function is calculated based upon said depth information. 

13. A method as in claim 12 comprising in addition the 27. A method as in claim 20 comprising, in addition, an 
step of conveying at least some of said image information improved method for applying paint system brush-oriented 
embodied as a video signal via an information bearing 55 functions, whereby for active/existing pixels within a brush 
medium. shape, which brush shape is often, but not necessarily, a 

14. A method as in claim 1 comprising, in addition, an bounding rectangle, the value of the image under each brush 
improved method for operator input of scaler information to pixel is replaced by some function of the brush pixel value, 
a computer system, wherein the improvement comprises or its neighborhood, and the value of the image under the 
interpretation of the input from a pliirality of transducers 60 brush pixel, or its neighborhood; and whereby prior to 
which are used in two distinct modes one of which is in a affecting said replacement, for each active/existing pixel 
coordinated fashion to specify values for a first set of within the brush shape, various conditions must be tested for 
variables and a second of which is in an anti-coordinated and met; which conditions include in any combination 
fashion to specify values for a second set of variables. criteria of values for color, protection, depth, obscuration, 

15. A method as in daim 1 comprising in addition the step 65 alpha, transparency and any other image plane present in a 
of conveying at least some of said information via a digital spedfic computer paint system into which said improvement 
information bearing medium. has been incorporated, for the brush pixel value, image pixel 
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value or some combination of the two; which results in a 
necessity to perform as many as PxC tests where P is defined 
as the number of pixels within said brush shape, and C is 
defined as the number of conditions to be met for any pixel 
within said brush shape to maintain active/existing status; 
wherein the improvement comprises: maintenance of said 
brush as an array of pixel values and maintenance of a list 
of brush pixel address indices; and whereby non-activc/non- 
existent pixels which are within the bounding shape do not 
appear on the list; and further manipulation of said list 
whereby when a pixel fails any one test which is critical said 
pixel is removed from said list and eliminated from consid- 
eration for subsequent tests of conditions to be met. 

28. A method as in claim 20 comprising, in addition, an 
improved method for applying paint system brush-oriented 
functions, wherein the improvement comprises that during 
execution of at least some paint function an image protection 
plane, of at least one bit, is maintained such that said image 
protection plane may be both written into and referenced 
during painting and whereby the action of the paint function 
on the image is adjusted based upon values abcady stored in 
said image protection plane. 

29. A method as in claim 20 comprising, in addition, an 
improved method for applying paint system brush-oriented 
functions, wherein the improvement comprises application 
of image processing fimctions, which in standard practice 
are applied by a program to an entire image or within a 
bounding shape, including contrast correction, color 
correction, gamma correction, gain, ofEset, and math and 
logical functions, to pixels imder a brush being used by an 
operator. 

30. A method for compatible stereoscopic broadcast com- 
prising: the broadcast of a two-dimensional image which 
may be displayed on a standard-type television display; 
additionally, the broadcast of depth information which is not 
visibly displayed on a standard television; the distortion of 
said two-dimensional image into two images, at least one of 
which has been modified by the method of claim 20 with 
respect to said two-dimensional image based upon said 
depth information; and, the stereoscopic display of said two 
images. 

31. A method as in claim 30 wherein said depth infonma- 
tion is encoded into a video blanking interval prior to the 
two-dimensional image which is to be modified based upon 
said depth information. 

32. A improved method for stereoscopic display of infor- 
mation created by the process of claim 20, wherein the 
improvement comprises display of said stereoscopic display 
as an anaglyphic display comprising a first image tinted by 
a first color and a second image tinted by a second color, and 
interleaving of segments tinted by said first color only with 
segments tinted by said second color only. 

33. A method as in claim 32 wherein distribution of those 
segments tinted by a first color only and those segments 
tinted by a second color only is varied on a frame-to-firame 
basis by a variable Spatial-Temporal Signature scheme. 

34. A method as in claim 32 comprising in addition the 
step of conveying at least some of said image information 
embodied as a video signal via an information bearing 
medium. 

35. A method as in claim 32 wherein said segments each 
comprise one video line. 

36. A method as in claim 20 wherein said depth informa- 
tion is referenced to determine if a selected painting function 
is to be apphed to a selected area of image information. 

37. A method as in claim 20 comprising in addition the 
step of conveying at least some of said information via a 
digital information bearing medium. 
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38. A method as in claim 30 comprising in addition the 
step of conveying at least some of said image information 
embodied as a video signal via an information bearing 
medium. 

5 39. A method for the creation of a stereoscopic display 
comprising the interleaving of at least two images at least 
one of which was created by the method of claim 20, 
wherein at least some of the interleaved segments are less 
than one video field. 

jQ 40. A product produced by the method of claim 39 and 
embodied as a video signal carried by an information 
bearing medium. 

41. An improved method as in claim 20 for displaying 
diagnostic and other medical images which can include 

j5 x-ray, CAT, NMR, endoscopic, photographic, video, com- 
puter or film images, wherein the improvement comprises 
the creation or modification of depth information to be 
incorporated into said medical images. 

42. A method for viewing sequentially interleaved bin- 
20 ocular information created by the method of claim 20 

comprising in addition to said method the viewing of said 
information by use of a set of glasses incorporating light 
sources which arc directed to each eye separately wherein 
the Ught sources for the left and right eye are alternated in 
25 such a manner that the light source is illuminated for the eye 
opposite to that for which an image being displayed is 
intended. 

43. An improved method for creating three-dimensional 
images by the method of claim 20 wherein the improvement 

3Q comprises that: for areas of ambiguous or uncertain depth, 
such as boundaries between objects separated in the depth 
direction, the depth value utilized is modulated by a random- 
noise -like fiinction. 

44. A method as in claim 20 wherein said image final 
35 information is three-dimensional. 

45. A method as in claim 20 wherein said image final 
information is stereoscopic. 

•46. A method as in claim 20 wherein the display of said 
final image information is effected by said stored image 
4Q depth information. 

47. A method as in claim 20 comprising in addition the 
creation or modification of depth information to be incor- 
porated into said final image information. 

48. A method as in claim 20 wherein at least some of said 
45 image depth information is non-planar. 

49. A method as in claim 48 wherein, in addition, each 
pixel of standard image information has associated with it a 
unique pixel of image depth information, and wherein said 
at least one function effects said image depth information 

5Q diiring its operation. 

50. A computer program stored on a computer readable 
medium comprising: 

a. instructions for performing image creation/ 
modification functions which include at least one fimc- 

55 tion available in standard computer paint systems 
including brush-oriented painting functions; and, 

b. instructions for storing image depth information in 
addition to image color information wherein said image 
depth information specifies other than a single plane 

60 perpendicular to the depth axis when depth excursions 
representing surface texture arc ehminated. 

51. A computer program as in claim 50 for creating virtual 
reality databases comprising in addition: 

c. instructions for deriving virtual reality shape informa- 
65 tion from said image depth information. 

52. A computer program as in claim 51 additionally 
comprising instructions to provide that: 
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a. virtual reality texture information is derived from the 
color information of said three-dimensional image; 

b. said program can operate repeatedly on a series of 
separate still images each of which is derived from a 
motion picture sequence; and, 

c. additional depth, position or shape information is 
automatically derived by computer analysis of the 
changes in geometry of elements contained in a mul- 
tiplicity of said still images. 

53. A computer program as in claim 51additionally com- 
prising: 

d. instructions for deriving virtual reality texture infor- 
mation from said image color information. 

54. A computer program as in claim 51, wherein said 
virtual reality is a videogame. 

55. A computer program as in claim 50 additionally 
comprising instructions to provide for the display of diag- 
nostic and other medical images which can include x-ray, 
CAT, ^3MR, endoscopic, photographic, video, computer or 
film images, wherein the improvement comprises the cre- 
ation or modification of depth information to be incorpo- 
rated into said medical images. 

56. An improved method for creating image information 
by using a computer image creation/modification system, 
wherein image creation/modification fimctions include at 
least one function available in standard computer paint 
systems including brush -oriented painting functions, 
wherein said improvement comprises storage of depth infor- 
mation in addition to standard image information and 
wherein an additional improvement comprises the ability to 
separately inhibit writing to at least one of a multiplicity of 
information planes which can include red, green, blue, color 
index, depth, obscuration, protection, alpha, transparency 
and any other image plane present in a specific computer 
paint system into which said additional improvement has 
been incorporated. 

57. A method as in claim 56 wherein an additional 
improvement comprises at least one function, such as 'paint- 
ing a sheet of glass% that updates depth information plane 
values without updating color information plane values. 

58. A method as in claim 56 wherein an additional 
improvement comprises at least one function, such as shape 
paint, that updates color information plane values without 
updating depth information plane values. 

59. An improved method for creating final image infor- 
mation by using a computer image creation/modification 
system, wherein image creation/modification functions 
include at least one function available in standard computer 
paint systems including brush-oriented painting functions, 
wherein said improvement comprises storage of image depth 
information in addition to image color information, and 
wherein said at least one fimction effects said image depth 
information during its operation, and wherein said at least 
one funaion produces dynamic visible feedback in the 
display of said final image information of its effect upon at 
least some of said image information during its operation. 

60. A method as in claim 59 for creating virtual reality 
databases whereby virtual reality shape information is 
derived from said image depth information. 

61. A method as in claim 60 comprising in addition the 
step of conveying at least some of said information embod- 
ied as a virtual reality database via a digital information 
bearing mediimi. 

62. A computer program as in claim 61, wherein said 
virtual reality is a videogame. 

63. A method as in claim 60 whereby, in addition, virtual 
reality texture information is derived from said image color 
information. 
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64. A method as in daim 63 for creating virtual reality 
databases whereby said method is repeatedly applied to a 
series of separate still images each of which is derived from 
a motion picture sequence. 
5 65. A method as in claim 64 wherein additional depth, 
position or shape information is automatically derived by 
computer analysis of the changes in geometry of elements 
contained in a multiplicity of said still images. 

66. A product produced by the method of claim 64 
10 comprising a virtual reality database and mediated by a 

digital information bearing medium. 

67. A computer program as in claim 66, wherein said 
virtual reality is a videogame. 

68. A computer program as in claim 64, wherein said 
15 virtual reality is a videogame. 

69. A method as in claim 59 wherein said image final 
information is three-dimensional. 

70. A method as in claim 59 wherein said image final 
information is stereoscopic. 

20 71. A method as in claim 59 wherein the display of said 
final image information is effected by said stored image 
depth information. 

72. A method as in claim 59 comprising in addition the 
creation or modification of depth information to be incor- 

25 porated into said final image information. 

73. A method as in claim 59 comprising in addition the 
step of conveying at least some of said information via an 
information bearing medium. 

74. A method as in claim 59 wherein said image depth 
30 information specifies other than a single plane perpendicular 

to the depth axis when depth excursions representing surface 
texture are eliminated. 

75. A method for modifying a pre-existing stereoscopic 
image by: 

35 a. creating from the information comprising said pre- 
existing stereoscopic image an image set of at least two 
images where stereoscopic depth is expressed as par- 
allax offsets between coherent features as represented 
in individual images of said image set; and, 

^ b. altering parallax shifts of said coherent image features 
within said image set. 

76. An improved method for creating image information 
by using a computer paint system, wherein painting func- 
tions include at least some function available in standard 
computer paint systems including brush-oriented painting 
and pen-oriented drawing functions, wherein said improve- 
ment comprises application of at least one paint function in 
such a manner that: 

a. those portions of the function which effect a first image 
are offset from those portions of the function which 
effect a second image; 

b. where said offset is related to the depth al which said 
paint function is to be applied to said images; 

55 c. where said first and said second image arc displayed in 
combination in an interleaved fashion; and, 
d. where the two offset portions of the visual representa- 
tion of said paint function are displayed in an inter- 
leaved fashion. 

60 77. An improved method for creating image information 
by using a computer image creation/modification system, 
wherein image creatioo/modification functions include at 
least one function available in standard computer paint 
systems including brush-oriented painting functions, 

65 wherein said improvement comprises storage of depth infor- 
mation in addition to standard image information and 
comprising, in addition, an improved method of presenting 
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a Graphic User Interface on a computer system, which 
Graphic User Interface comprises at least one of a standard 
set of elements common to Graphic User Interfaces includ- 
ing: buttons, menus, palettes, slidebars, check boxes, frames 
and windows; where the improvement comprises displaying 
at least one of said elements in stereo scopic format. 

78. A method as in claim 77 wherein said Graphic User 
Interface is employed as an interface to an animation system. 

79. A method as in claim 77 wherein said Graphic User 
Interface is employed as an interface to a microprocessor- 
based windowing system. 

80. A method as in claim 79 wherein said Graphic User 
Interface is employed in addition as an interface to visual 
programming language software. 

81. A method as in claim 77 comprising in addition that 
said at least one of said elements in stereoscopic format 
comprises in addition a stereoscopic drop shadow. 

82. An improved method for creating image information 
by using a computer image, creation/modification system, 
wherein image creation/modification functions include at 
least one function available in standard computer paint 
systems including brush-oriented painting functions, 
wherein said improvement comprises storage of depth infor- 
mation in addition to standard image information and 
comprising, in addition, an improved method for operator 
input of scaler information to a computer system, wherein 
the improvement comprises interpretation of the input from 
a pltu-ality of transducers which are used in two distinct 
modes one of which is in a coordinated fashion to specify 
values for a first set of variables and a second of which is in 
an anti-coordinated fashion to specify values for a second set 
of variables, and at least some of said variables are input to 
control at least some of said image creation/modification 
functions. 

83. A method as in claim 6 providing a third mode of 
operation which mode is an uncoordinated mode whereby at 
least one first transducer receives null input and at least one 
second transducer receives input resulting in specification of 
a third set of variables. 

84. A method as in claim 6 wherein the plurality of 
transducers comprises at least two translation transducers 
which when used in a coordinated fashion specify transla- 
tion and when used in an anti-coordinated fashion specify 
rotation. 

85. A method as in claim 82 comprising, in addition, an 
improved method for interpreting input from a device for the 
operator input of scaler information to a computer system, 
whereby the improvement comprises accumulating the input 
in one direction and ignoring input in the other direction. 

86. A method as in claim 82, wherein the method is to be 
used to create/modify a mullipUcily of images and it is 
desired to be able to display any of said multiplicity of 
images for creation, modification, review or presentation, 
said method comprising, in addition, an improved method 
for controlling the presentation of a multiplicity of ordered 
images as presented on a computer screen, wherein the 
improvement comprises interpreting input from a device for 
operator input of scaler information to a computer, such that 
the ntunber of images skipped over between the currently 
displayed image and the next image displayed is directly 
dependent upon input from the device by utilization of a 
page flip paradigm. 

87. An improved method for creating image information 
by using a computer image creation/modification system, 
wherein image creation/modification functions include at 
least one function available in standard computer paint 
systems including brush -oriented painting functions, 
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wherein said improvement comprises storage of depth infor- 
mation in addition to standard image information and 
comprising, in addition, an improved method for operator 
input of scaler information to a computer system, wherein 
5 the improvement comprises interpretation of input from a 
plurality of rotational transducers at least one of which is 
used in at least two modes to specify in each of said modes 
a different variable of information. 

88. A method as in claim 87 wherein the plurality of 
rotational transducers comprises two balls. 

89. A method as in claim 87 wherein the plurality of 
rotational transducers comprises four balls at the comers of 
a square and one pivot ball in the center and where at least 
the four corner balk provide active input data. 

90. A computer program stored on a computer readable 
medium comprising: 

a. instructions for performing computer paint system 
functions which include at least one function available 
in standard computer paint systems including brush- 
oriented painting functions, used for the creation of 
three-dimensional image information; 

b. instructions for maintaining a plurality of image infor- 
mation memories; and, 

c. instructions for applying at least one paint function to 
25 at least two of said image information memories, 

wherein the application of said at least one paint 
function in a first of said image information memories 
is offset with respect to the application of the same 
paint function in a second of said image information 
3Q memories. 

91. A computer program as in claim 90 additionally 
comprising: 

a. instructions for uniquely associating said plurality of 
image information memories each with at least one of 

35 a plurality of image information elements; 

b. instructions for creating left and right images by 
applying parallax ofEsets to at least some of said plu- 
rality of image information elements; and, 

c. instructions for determining said parallax offsets as a 
40 function of depth information associated with at least 

some of said image information elements. 

92. A computer program as in claim 91 wherein said 
offsets may be adjusted by a fraction of a pixel for at least 
one image information memory value by further comprising 

45 instructions for applying interpolation to apportion said 
image information value to more than one image informa- 
tion display pixel. 

93. An improved method for creating final image infor- 
mation by using a computer image creation/modification 

50 system, wherein image creation/modification functions 
include at least one function available in standard computer 
paint systems including brush-oriented painting functions, 
wherein said improvement comprises storage of image depth 
information in addition to image color information, and 

55 wherein each pixel of image color information has associ- 
ated with it a unique pixel of image depth information, and 
wherein said at least one function produces dynamic visible 
feedback in the display of said final image information of its 
effect upon at least some of said image information during 

60 its operation. 

94. A method as in claim 93 wherein said image final 
information is three-diminsional - 

95. A method as in claim 93 wherein said image final 
information is stereoscopic. 

65 96. A method as in claim 93 wherein the display of said 
final image information is effected by said stored image 
depth information. 
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97. A method as in claim 93 comprising in addition the brush pixel, or its neighborhood; and whereby prior to 
creation or modification of depth information to be incor- affecting said replacement, for each active/existiDg pixel 
porated into said final image information. within the brush shape, various conditions must be tested for 

98. A method as in claim 93 wherein said image depth and met; which conditions include in any combination 
information specifies other than a single plane perpendicular 5 criteria of values for color, protection, depth, obscuration, 
to the depdi axis when depth excursions representing surface alpha, transparency and any other image plane present in a 
texture are eliminated. specific computer paint system into which said improvement 

99. A method as in claim 93 wherein obscuring of one ^^^^ incorporated, for the brush pixel value, image pixel 
image information element by another is based on said depth ^^^^ ^^^^ combination of the two; which results in a 

information. . , . , . , necessity to perform as many as PxC tests where P is defined 

100. A method as m claim 93 wherem an additional ^ ^^^^^ ^^^^ ^-^^ ^ 

miprovement comprises at least one function to pamt with a u c j *- * u *r • i 

. ^ , , i . . , 1 . 1 • defined as the number of conditions to be met for any pixel 

brush that applies both color value to a colorimage memory ... ... , . ^ . . _*-/ •,• 

plane and depth value to a depth image memory plane. ■ ^^'^ ^"^^ ^^P^ ^° active/existing status; 

101. A method as in claim 93 wherein said depth infor- 15 "^^^'^"^ improvement compnscs: mamtenance of said 
mation is sub-divided into depth tag and depth value infor- t)^sh as an anay of pixel values and mamtenance of a list 
mation and the depth specification to be used is derived as of brush pixel address mdices; and whereby non-acUve/non- 
a function of the depth tag and depth value. existent pixels which are withm the bounding shape do not 

102. A method as in claim 93 wherein an additional appear on the Ust; and further manipulation of said list 
improvement comprises that for at least one image process- 20 whereby when a pixel fails any one test which is critical said 
ing function a set of pixels input to said image processing pixel is removed from said fist and eUminated from consid- 
function and a set of pixels effected by output of said image eration for subsequent tests of conditions to be met. 
processing function are dependent upon depth information 107. A method as in claim 93 comprising, in addition, an 
associated with at least some of said input and effected improved method for applying paint system brush-oriented 
pixels. 25 functions, wherein the improvement comprises that during 

103. A method as in claim 93 wherein an additional execution of at least some paint function an, image pro tec- 
improvement comprises inclusion in said created image tion plane, of at least one bit, is maintained such that said 
information of shadows cast by at least one first element image protection plane may be both written into and refer- 
onto at least one second element whereby a shadow casting enced during paintmg and whereby the action of the paint 
function is calculated based upon said depth information. 30 function on the image is adjusted based upon values already 

104. A method as in claim 93 wherein an additional stored in said image protection plane. 

improvement comprises at least one function, such as *paint- 108. A method as in claim 93 comprising, in addition, an 

ing a sheet of glass*, that updates depth information plane improved method for applying paint system brush-oriented 

values without updating color information plane values. functions, wherein the improvement comprises appUcation 

105. A method as in claim 93 wherein an additional 35 of image processing functions, which in standard practice 
improvement comprises at least one function, such as shape are applied by a program to an entire image or within a 
paint, that updates color information plane values without bounding shape, including contrast correction, color 
updating depth information plane values. correction, gamma correction, gain, offeet, and math and 

106. A method as in claim 93 comprising, in addition, an logical functions, to pixels under a brush being used by an 
improved method for applying paint system brush-oriented 40 operator. 

functions, whereby for active/existing pixels within a brush 109. A method as in claim 93 comprising in addition the 

shape, which brush shape is often, but not necessarily, a step of conveying at least some of said information via an 

bounding rectangle, the value of the image under each brush information bearing medium, 
pixel is replaced by some function of the brush pixel value, 

or its neighborhood, and the value of the image under the ***** 
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